# About Unstructured
Source: https://docs.unstructured.io/about



<Frame>
  <img alt="Unstructured conceptual data flow" />
</Frame>

Unstructured provides a platform and tools to ingest and process your unstructured documents for:

* [Enhancing retrieval-augmented generation (RAG)](https://unstructured.io/blog/rag-whitepaper): RAG boosts AI accuracy and relevance by working with the data that's most important to you, providing results that are more current, focused, and meaningful to your queries and tasks.
* [Fueling agentic AI](https://unstructured.io/problems-we-solve#powering-agentic-ai): Agentic AI acts like virtual teammates that can plan, decide, and take action on their own to get things done on your behalf, freeing you up for bigger challenges.

We empower your organization to take full advantage of RAG and agentic AI opportunities through:

* [Rapid prototyping](https://unstructured.io/problems-we-solve#rapid-prototyping): Quickly iterating on AI experiences powered by enterprise data.
* [Replacing do-it-yourself data transformation pipelines](https://unstructured.io/problems-we-solve#replacing-diy-pipelines): Building your own data transformation pipelines is time-consuming and error-prone.
* [Unifying data silos](https://unstructured.io/problems-we-solve#unifying-data-silos): Eliminating data barriers and creating shared GenAI data layers.
* [Continuous data delivery](https://unstructured.io/problems-we-solve#continuous-data-delivery): Scheduling jobs, routing documents, and optimizing every task.
* [Business process automation](https://unstructured.io/problems-we-solve#business-process-automation): Report generation, automatic responses, decision-making, content analysis, and more!

This 60-second video describes more about what Unstructured does and its benefits (no sound):

<iframe title="YouTube video player" />

This 40-second video demonstrates a simple use case that Unstructured helps solve (no sound):

<iframe title="YouTube video player" />

This 60-second video shows why using Unstructured is preferable to building your own similar solution:

<iframe title="YouTube video player" />

You can use Unstructured through a user interface (UI), an API, or both.

<Frame>
  <img alt="Unstructured user interface example" />
</Frame>

<Frame>
  <img alt="Unstructured API example" />
</Frame>

## <Icon icon="money-check-dollar" />    Pricing

Unstructured offers different account types with different pricing plans:

* <Icon icon="person" />  **Let's Go** and **Pay-As-You-Go** - A single user, with a single workspace, hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="building" />  **Business** - Multiple users and workspaces, with three options:

  * <Icon icon="people-group" />  **Business SaaS** - Hosted alongside other accounts on Unstructured's cloud infrastructure.
  * <Icon icon="shield-halved" />  **Dedicated instance** - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control.
  * <Icon icon="shield" />  **In-VPC** - Hosted within your own VPC on your own cloud infrastructure.

  **Business** accounts also allow for robust customization of Unstructured's features for your unique needs.

For more details, see the [Unstructured Pricing](https://unstructured.io/pricing) page.

To upgrade your account from **Let's Go** or **Pay-As-You-Go** to **Business**,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Some of these plans have billing details that are determined on a per-page basis.

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

## <Icon icon="question" />    Questions? Need help?

* For general questions about Unstructured's product offerings, email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
* For Unstructured technical support, see [Requesting support](/support/request).


# Create destination connector
Source: https://docs.unstructured.io/api-reference/api/destination/create-destination

POST /api/v1/destinations/
Create a new destination connector using the provided configuration.

## Body

<ParamField type="string">
  Destination connector name.
</ParamField>

<ParamField type="string">
  Connector type. One of: `azure`, `astradb`, `azure_ai_search`, `couchbase`, `databricks_volumes`, `databricks_volume_delta_tables`, `delta_table`, `elasticsearch`, `gcs`, `kafka-cloud`, `milvus`, `mongodb`, `motherduck`, `neo4j`, `onedrive`, `opensearch`, `pinecone`, `postgres`, `redis`, `qdrant-cloud`, `s3`, `snowflake`, `teradata`, `weaviate-cloud`, `ibm_watsonx_s3`.
</ParamField>

<ParamField type="object">
  Connector configuration. Required fields vary by connector `type`.

  For the specific settings to include for a connector, see the entry for that connector in [Destinations](/api-reference/workflow/destinations/overview).
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the destination connector.
</ResponseField>

<ResponseField name="name" type="string">
  Destination connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.

  For the specific settings to include for a specific connector, see the entry for that connector in [Destinations](/api-reference/workflow/destinations/overview).
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "name": "My S3 Destination",
      "type": "s3",
      "config": {
        "remote_url": "s3://my-bucket/output/",
        "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      }
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.destinations.create_destination(
      request=CreateDestinationRequest(
          create_destination_connector=CreateDestinationConnector(
              name="My S3 Destination",
              type="s3",
              config={
                  "remote_url": "s3://my-bucket/output/",
                  "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
                  "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
              },
          )
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  async def create_destination():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.destinations.create_destination_async(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="My S3 Destination",
                  type="s3",
                  config={
                      "remote_url": "s3://my-bucket/output/",
                      "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
                      "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
                  },
              )
          )
      )
      print(response)

  asyncio.run(create_destination())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
    "name": "My S3 Destination",
    "type": "s3",
    "config": {
      "remote_url": "s3://my-bucket/output/",
      "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "created_at": "2026-01-01T00:00:00Z",
    "updated_at": null
  }
  ```
</ResponseExample>


# Create destination connection check
Source: https://docs.unstructured.io/api-reference/api/destination/create-destination-connection-check

POST /api/v1/destinations/{destination_id}/connection-check
Test a destination connection by creating a connection check for the specified destination connector.

The Python SDK does not support testing destination connectors.

## Path parameters

<ParamField type="string">
  The unique identifier of the destination connector.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the connection check.
</ResponseField>

<ResponseField name="status" type="string">
  Connection check status. One of: `SCHEDULED`, `SUCCESS`, `FAILURE`.
</ResponseField>

<ResponseField name="reason" type="string">
  Failure reason, if applicable.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the check was initiated.
</ResponseField>

<ResponseField name="reported_at" type="string">
  ISO 8601 timestamp when the check result was reported.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c/connection-check" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
    "status": "SCHEDULED",
    "reason": null,
    "created_at": "2026-01-01T00:00:00Z",
    "reported_at": null
  }
  ```
</ResponseExample>


# Delete destination connector
Source: https://docs.unstructured.io/api-reference/api/destination/delete-destination

DELETE /api/v1/destinations/{destination_id}
Delete a specific destination connector by its ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the destination connector.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request DELETE \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteDestinationRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.destinations.delete_destination(
      request=DeleteDestinationRequest(
          destination_id="d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteDestinationRequest

  async def delete_destination():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.destinations.delete_destination_async(
          request=DeleteDestinationRequest(
              destination_id="d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
          )
      )
      print(response)

  asyncio.run(delete_destination())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Destination endpoints
Source: https://docs.unstructured.io/api-reference/api/destination/destination-apis



Manage the destination connectors that receive processed document data from your Unstructured workflows. Create and configure connections to storage systems, verify connectivity, and keep connectors up to date.

| Name                                                                                                      | Method   | Path                                                     |
| --------------------------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------- |
| [List destination connectors](/api-reference/api/destination/list-destinations)                           | `GET`    | `/api/v1/destinations/`                                  |
| [Get destination connector](/api-reference/api/destination/get-destination)                               | `GET`    | `/api/v1/destinations/{destination_id}`                  |
| [Create destination connector](/api-reference/api/destination/create-destination)                         | `POST`   | `/api/v1/destinations/`                                  |
| [Update destination connector](/api-reference/api/destination/update-destination)                         | `PUT`    | `/api/v1/destinations/{destination_id}`                  |
| [Delete destination connector](/api-reference/api/destination/delete-destination)                         | `DELETE` | `/api/v1/destinations/{destination_id}`                  |
| [Get destination connection check](/api-reference/api/destination/get-destination-connection-check)       | `GET`    | `/api/v1/destinations/{destination_id}/connection-check` |
| [Create destination connection check](/api-reference/api/destination/create-destination-connection-check) | `POST`   | `/api/v1/destinations/{destination_id}/connection-check` |


# Get destination connector
Source: https://docs.unstructured.io/api-reference/api/destination/get-destination

GET /api/v1/destinations/{destination_id}
Retrieve detailed information for a specific destination connector.

## Path parameters

<ParamField type="string">
  The unique identifier of the destination connector.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the destination connector.
</ResponseField>

<ResponseField name="name" type="string">
  Destination connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetDestinationRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.destinations.get_destination(
      request=GetDestinationRequest(
          destination_id="d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetDestinationRequest

  async def get_destination():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.destinations.get_destination_async(
          request=GetDestinationRequest(
              destination_id="d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
          )
      )
      print(response)

  asyncio.run(get_destination())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
    "name": "My S3 Destination",
    "type": "s3",
    "config": {
      "remote_url": "s3://my-bucket/output/",
      "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "created_at": "2026-01-01T00:00:00Z",
    "updated_at": "2026-01-15T12:00:00Z"
  }
  ```
</ResponseExample>


# Get destination connection check
Source: https://docs.unstructured.io/api-reference/api/destination/get-destination-connection-check

GET /api/v1/destinations/{destination_id}/connection-check
Retrieve the most recent connection check for the destination connector.

The Python SDK does not support getting information about the most recent connector check for a destination connector.

## Path parameters

<ParamField type="string">
  The unique identifier of the destination connector.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the connection check.
</ResponseField>

<ResponseField name="status" type="string">
  Connection check status. One of: `SCHEDULED`, `SUCCESS`, `FAILURE`.
</ResponseField>

<ResponseField name="reason" type="string">
  Failure reason, if applicable.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the check was initiated.
</ResponseField>

<ResponseField name="reported_at" type="string">
  ISO 8601 timestamp when the check result was reported.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c/connection-check" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
    "status": "SUCCESS",
    "reason": null,
    "created_at": "2026-01-01T00:00:00Z",
    "reported_at": "2026-01-01T00:00:05Z"
  }
  ```
</ResponseExample>


# List destination connectors
Source: https://docs.unstructured.io/api-reference/api/destination/list-destinations

GET /api/v1/destinations/
Retrieve a list of available destination connectors.

## Query parameters

<ParamField type="string">
  Filter by connector type. One of: `azure`, `astradb`, `azure_ai_search`, `couchbase`, `databricks_volumes`, `databricks_volume_delta_tables`, `delta_table`, `elasticsearch`, `gcs`, `kafka-cloud`, `milvus`, `mongodb`, `motherduck`, `neo4j`, `onedrive`, `opensearch`, `pinecone`, `postgres`, `redis`, `qdrant-cloud`, `s3`, `snowflake`, `teradata`, `weaviate-cloud`, `ibm_watsonx_s3`.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the destination connector.
</ResponseField>

<ResponseField name="name" type="string">
  Destination connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListDestinationsRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.destinations.list_destinations(
      request=ListDestinationsRequest()
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListDestinationsRequest

  async def list_destinations():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.destinations.list_destinations_async(
          request=ListDestinationsRequest()
      )
      print(response)

  asyncio.run(list_destinations())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  [
    {
      "id": "d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
      "name": "My S3 Destination",
      "type": "s3",
      "config": {
        "remote_url": "s3://my-bucket/output/",
        "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      },
      "created_at": "2026-01-01T00:00:00Z",
      "updated_at": "2026-01-15T12:00:00Z"
    }
  ]
  ```
</ResponseExample>


# Update destination connector
Source: https://docs.unstructured.io/api-reference/api/destination/update-destination

PUT /api/v1/destinations/{destination_id}
Update the configuration of an existing destination connector.

You must specify all of the settings for the connector, even for settings that are not changing.

You can change any of the connector's settings except for its `name`and `type`.

## Path parameters

<ParamField type="string">
  The unique identifier of the destination connector.
</ParamField>

## Body

<ParamField type="string">
  New destination connector name.
</ParamField>

<ParamField type="object">
  Updated connector configuration. Fields vary by connector type.

  For the specific settings to include for a specific connector, see the entry for that connector in [Destinations](/api-reference/workflow/destinations/overview).
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the destination connector.
</ResponseField>

<ResponseField name="name" type="string">
  Destination connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request PUT \
    --url "${UNSTRUCTURED_API_URL}/api/v1/destinations/d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "name": "My Updated S3 Destination",
      "config": {
        "remote_url": "s3://my-new-bucket/output/",
        "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      }
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateDestinationRequest
  from unstructured_client.models.shared import UpdateDestinationConnector

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.destinations.update_destination(
      request=UpdateDestinationRequest(
          destination_id="d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
          update_destination_connector=UpdateDestinationConnector(
              name="My Updated S3 Destination",
              config={
                  "remote_url": "s3://my-new-bucket/output/",
                  "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
                  "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
              },
          )
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateDestinationRequest
  from unstructured_client.models.shared import UpdateDestinationConnector

  async def update_destination():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.destinations.update_destination_async(
          request=UpdateDestinationRequest(
              destination_id="d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
              update_destination_connector=UpdateDestinationConnector(
                  name="My Updated S3 Destination",
                  config={
                      "remote_url": "s3://my-new-bucket/output/",
                      "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
                      "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
                  },
              )
          )
      )
      print(response)

  asyncio.run(update_destination())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "d1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
    "name": "My Updated S3 Destination",
    "type": "s3",
    "config": {
      "remote_url": "s3://my-new-bucket/output/",
      "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    },
    "created_at": "2026-01-01T00:00:00Z",
    "updated_at": "2026-04-29T10:00:00Z"
  }
  ```
</ResponseExample>


# Cancel job
Source: https://docs.unstructured.io/api-reference/api/job/cancel-job

POST /api/v1/jobs/{job_id}/cancel
Cancel the specified job.

## Path parameters

<ParamField type="string">
  The unique identifier of the job.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e/cancel" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CancelJobRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.cancel_job(
      request=CancelJobRequest(
          job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CancelJobRequest

  async def cancel_job():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.cancel_job_async(
          request=CancelJobRequest(
              job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
          )
      )
      print(response)

  asyncio.run(cancel_job())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Create job
Source: https://docs.unstructured.io/api-reference/api/job/create-job

POST /api/v1/jobs/
Create a new on-demand workflow job using either a template or custom DAG.

An *on-demand workflow job* is a job that takes one or more local files only as input, and whose temporary workflow exists only for the duration of the job's run

<Note>
  To run a workflow that was already created at some point in the past and still exists (also known as a *long-lived workflow*), see the [run workflow endpoint](/api-reference/api/workflow/run-workflow) instead.
  To run a workflow that takes files and data from remote locations as input (instead of local files), do the following instead:

  1. [Create a source connector](/api-reference/api/source/create-source) to the remote source locations.
  2. [Create a destination connector](/api-reference/api/destination/create-destination) to the remote destination location.
  3. [Create a long-lived workflow](/api-reference/api/workflow/create-workflow) that uses this specific source connector and destination connector.
  4. [Run this long-lived workflow manually](/api-reference/api/workflow/run-workflow), if you have not already created the workflow to run on a schedule.
</Note>

## Body

<ParamField type="string">
  Job configuration data.

  * To use a workflow template for a job, include a `template_id` field that specifies the unique ID of the workflow template. For more information, see [list templates](/api-reference/api/template/list-templates).
  * To use a custom workflow definition for a job, include a `job_nodes` field that specifies the settings for the job's workflow nodes. For instructions, see [Workflow Nodes](/api-reference/workflow/nodes/overview).
</ParamField>

<ParamField type="array">
  Input files to process. Upload as multipart/form-data, in the following format:

  ```
  --form "input_files=@</full/path/to/local/filename.extension>;filename=<filename.extension>;type=<local-file-media-type>" \
  --form "input_files=@</full/path/to/local/filename.extension>;filename=<filename.extension>;type=<local-file-media-type>" # For each additional file to be uploaded.
  ```

  For more information, see [Unstructured API Quickstart - On-Demand Jobs](/api-reference/quickstart).
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the job.
</ResponseField>

<ResponseField name="workflow_id" type="string">
  Unique identifier of the workflow that created this job.
</ResponseField>

<ResponseField name="workflow_name" type="string">
  Name of the workflow that created this job.
</ResponseField>

<ResponseField name="status" type="string">
  Job status. One of: `SCHEDULED`, `IN_PROGRESS`, `COMPLETED`, `STOPPED`, `FAILED`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the job was created.
</ResponseField>

<ResponseField name="runtime" type="string">
  ISO 8601 duration of the job run.
</ResponseField>

<ResponseField name="input_file_ids" type="array">
  IDs of input files for this job.
</ResponseField>

<ResponseField name="output_node_files" type="array">
  Output file metadata objects. Each object includes `node_id`, `file_id`, `node_type`, and `node_subtype`.
</ResponseField>

<ResponseField name="job_type" type="string">
  Job type. Default: `ephemeral`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --form "request_data={\"template_id\":\"f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c\"}" \
    --form "input_files=@/path/to/document.pdf"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateJobRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.create_job(
      request=CreateJobRequest(
          request_data="{\"template_id\":\"f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c\"}"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateJobRequest

  async def create_job():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.create_job_async(
          request=CreateJobRequest(
              request_data="{\"template_id\":\"f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c\"}"
          )
      )
      print(response)

  asyncio.run(create_job())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
    "workflow_id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
    "workflow_name": "My ETL Workflow",
    "status": "SCHEDULED",
    "created_at": "2026-01-01T00:00:00Z",
    "runtime": null,
    "input_file_ids": ["document.pdf"],
    "output_node_files": null,
    "job_type": "ephemeral"
  }
  ```
</ResponseExample>


# Download job output
Source: https://docs.unstructured.io/api-reference/api/job/download-job-output

GET /api/v1/jobs/{job_id}/download
Download the output of a job from a workflow.

<Note>
  This applies only to jobs that use a workflow with a local source and a local destination.
</Note>

To download a file's output from the last workflow node in a job run:

* Do not specify a node\_id argument. The last workflow node will be used by default.
* Use the [get job endpoint](/api-reference/api/job/get-job) to get the job's details. The file's Unstructured ID will be in the response's `input_file_ids` array.

To download a file's output from a specific workflow node in a job run:

* Use the [get job endpoint](/api-reference/api/job/get-job) to get the job's details:
  * The file's ID will be in the `file_id` field in the response's `output_node_files` array
  * The node's ID will be in the `node_id` field in the response's `output_node_files` array

Currently, you cannot use Unstructured Pipelines to download a file from a job that uses a workflow with a local source and a local destination.

## Path parameters

<ParamField type="string">
  The unique identifier of the job.
</ParamField>

## Query parameters

<ParamField type="string">
  The ID of the output file to download.
</ParamField>

<ParamField type="string">
  Filter by the node that produced the output file. The default is the last workflow node.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e/download?file_id=output-001.json" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --output output-001.json
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DownloadJobOutputRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.download_job_output(
      request=DownloadJobOutputRequest(
          job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
          file_id="output-001.json"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DownloadJobOutputRequest

  async def download_job_output():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.download_job_output_async(
          request=DownloadJobOutputRequest(
              job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
              file_id="output-001.json"
          )
      )
      print(response)

  asyncio.run(download_job_output())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  <binary file content>
  ```
</ResponseExample>


# Get job
Source: https://docs.unstructured.io/api-reference/api/job/get-job

GET /api/v1/jobs/{job_id}
Retrieve basic information for a specific job.

To get current processing information about a job, use the [get job details endpoint](/api-reference/api/job/get-job-details).

## Path parameters

<ParamField type="string">
  The unique identifier of the job.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the job.
</ResponseField>

<ResponseField name="workflow_id" type="string">
  Unique identifier of the workflow that created this job.
</ResponseField>

<ResponseField name="workflow_name" type="string">
  Name of the workflow that created this job.
</ResponseField>

<ResponseField name="status" type="string">
  Job status. One of: `SCHEDULED`, `IN_PROGRESS`, `COMPLETED`, `STOPPED`, `FAILED`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the job was created.
</ResponseField>

<ResponseField name="runtime" type="string">
  ISO 8601 duration of the job run.
</ResponseField>

<ResponseField name="input_file_ids" type="array">
  IDs of input files for this job.
</ResponseField>

<ResponseField name="output_node_files" type="array">
  Output file metadata objects. Each object includes `node_id`, `file_id`, `node_type`, and `node_subtype`.
</ResponseField>

<ResponseField name="job_type" type="string">
  Job type. Default: `ephemeral`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetJobRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.get_job(
      request=GetJobRequest(
          job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetJobRequest

  async def get_job():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.get_job_async(
          request=GetJobRequest(
              job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
          )
      )
      print(response)

  asyncio.run(get_job())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
    "workflow_id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
    "workflow_name": "My ETL Workflow",
    "status": "COMPLETED",
    "created_at": "2026-01-01T00:00:00Z",
    "runtime": "PT2M30S",
    "input_file_ids": null,
    "output_node_files": [
      {
        "node_id": "n1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
        "file_id": "output-001.json",
        "node_type": "destination",
        "node_subtype": "s3"
      }
    ],
    "job_type": "ephemeral"
  }
  ```
</ResponseExample>


# Get job details
Source: https://docs.unstructured.io/api-reference/api/job/get-job-details

GET /api/v1/jobs/{job_id}/details
Retrieve current processing details for a specific job.

## Path parameters

<ParamField type="string">
  The unique identifier of the job.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the job.
</ResponseField>

<ResponseField name="processing_status" type="string">
  Processing status. One of: `SCHEDULED`, `IN_PROGRESS`, `SUCCESS`, `COMPLETED_WITH_ERRORS`, `STOPPED`, `FAILED`.
</ResponseField>

<ResponseField name="node_stats" type="array">
  Per-node processing statistics. Each object includes `node_name`, `node_type`, `node_subtype`, `ready`, `in_progress`, `success`, and `failure` counts.
</ResponseField>

<ResponseField name="message" type="string">
  Additional status message, if available.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e/details" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetJobDetailsRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.get_job_details(
      request=GetJobDetailsRequest(
          job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetJobDetailsRequest

  async def get_job_details():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.get_job_details_async(
          request=GetJobDetailsRequest(
              job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
          )
      )
      print(response)

  asyncio.run(get_job_details())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
    "processing_status": "SUCCESS",
    "node_stats": [
      {
        "node_name": "partition",
        "node_type": "partition",
        "node_subtype": "auto",
        "ready": 0,
        "in_progress": 0,
        "success": 42,
        "failure": 0
      }
    ],
    "message": null
  }
  ```
</ResponseExample>


# Get job failed files
Source: https://docs.unstructured.io/api-reference/api/job/get-job-failed-files

GET /api/v1/jobs/{job_id}/failed-files
Retrieve the list of any failed files for a specific job, and why those files failed.

## Path parameters

<ParamField type="string">
  The unique identifier of the job.
</ParamField>

## Response

<ResponseField name="failed_files" type="array">
  List of failed files. Each object includes `document` (the file path or identifier) and `error` (the failure reason).
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e/failed-files" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetJobFailedFilesRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.get_job_failed_files(
      request=GetJobFailedFilesRequest(
          job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetJobFailedFilesRequest

  async def get_job_failed_files():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.get_job_failed_files_async(
          request=GetJobFailedFilesRequest(
              job_id="b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
          )
      )
      print(response)

  asyncio.run(get_job_failed_files())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "failed_files": [
      {
        "document": "s3://my-bucket/input/corrupted-file.pdf",
        "error": "Failed to parse document: unexpected end of file"
      }
    ]
  }
  ```
</ResponseExample>


# Job endpoints
Source: https://docs.unstructured.io/api-reference/api/job/job-apis



Track and control the processing jobs that execute your workflows. Trigger runs, monitor status, download results, and inspect any files that failed during processing.

| Name                                                                | Method | Path                                 |
| ------------------------------------------------------------------- | ------ | ------------------------------------ |
| [List jobs](/api-reference/api/job/list-jobs)                       | `GET`  | `/api/v1/jobs/`                      |
| [Get job](/api-reference/api/job/get-job)                           | `GET`  | `/api/v1/jobs/{job_id}`              |
| [Create job](/api-reference/api/job/create-job)                     | `POST` | `/api/v1/jobs/`                      |
| [Cancel job](/api-reference/api/job/cancel-job)                     | `POST` | `/api/v1/jobs/{job_id}/cancel`       |
| [Download job output](/api-reference/api/job/download-job-output)   | `GET`  | `/api/v1/jobs/{job_id}/download`     |
| [Get job details](/api-reference/api/job/get-job-details)           | `GET`  | `/api/v1/jobs/{job_id}/details`      |
| [Get job failed files](/api-reference/api/job/get-job-failed-files) | `GET`  | `/api/v1/jobs/{job_id}/failed-files` |

A workflow job is created automatically whenever:

* A workflow runs on a schedule. For more details see [Create workflow](/api-reference/api/workflow/create-workflow).
* You run a workflow manually. For more details see [Run workflow](/api-reference/api/workflow/run-workflow).


# List jobs
Source: https://docs.unstructured.io/api-reference/api/job/list-jobs

GET /api/v1/jobs/
Retrieve a list of jobs with optional filtering.

## Query parameters

<ParamField type="string">
  Filter by workflow ID.
</ParamField>

<ParamField type="string">
  Filter by job status.

  | Value         | Description                |
  | ------------- | -------------------------- |
  | `SCHEDULED`   | Job is queued to run.      |
  | `IN_PROGRESS` | Job is currently running.  |
  | `COMPLETED`   | Job finished successfully. |
  | `STOPPED`     | Job was manually stopped.  |
  | `FAILED`      | Job failed to complete.    |
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the job.
</ResponseField>

<ResponseField name="workflow_id" type="string">
  Unique identifier of the workflow that created this job.
</ResponseField>

<ResponseField name="workflow_name" type="string">
  Name of the workflow that created this job.
</ResponseField>

<ResponseField name="status" type="string">
  Job status.

  | Value         | Description                |
  | ------------- | -------------------------- |
  | `SCHEDULED`   | Job is queued to run.      |
  | `IN_PROGRESS` | Job is currently running.  |
  | `COMPLETED`   | Job finished successfully. |
  | `STOPPED`     | Job was manually stopped.  |
  | `FAILED`      | Job failed to complete.    |
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the job was created.
</ResponseField>

<ResponseField name="runtime" type="string">
  ISO 8601 duration of the job run.
</ResponseField>

<ResponseField name="input_file_ids" type="array">
  IDs of input files for this job.
</ResponseField>

<ResponseField name="output_node_files" type="array">
  Output file metadata objects. Each object includes `node_id`, `file_id`, `node_type`, and `node_subtype`.
</ResponseField>

<ResponseField name="job_type" type="string">
  Job type. Default: `ephemeral`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListJobsRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.jobs.list_jobs(
      request=ListJobsRequest()
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListJobsRequest

  async def list_jobs():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.jobs.list_jobs_async(
          request=ListJobsRequest()
      )
      print(response)

  asyncio.run(list_jobs())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  [
    {
      "id": "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
      "workflow_id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
      "workflow_name": "My ETL Workflow",
      "status": "COMPLETED",
      "created_at": "2026-01-01T00:00:00Z",
      "runtime": "PT2M30S",
      "input_file_ids": null,
      "output_node_files": [
        {
          "node_id": "n1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
          "file_id": "output-001.json",
          "node_type": "destination",
          "node_subtype": "s3"
        }
      ],
      "job_type": "ephemeral"
    }
  ]
  ```
</ResponseExample>


# Create notification channel
Source: https://docs.unstructured.io/api-reference/api/notification/create-notification-channel

POST /api/v1/notifications/channels
Create a new delivery channel for receiving platform event notifications.

<Warning>
  The `secret` field is write-only and is not returned by the API after creation. To change it later, use the [update channel endpoint](/api-reference/api/notification/update-notification-channel) instead.
</Warning>

## Body

<ParamField type="string">
  Channel type: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ParamField>

<ParamField type="array">
  Event types to subscribe to.

  | Value             | Description                                           |
  | ----------------- | ----------------------------------------------------- |
  | `job.scheduled`   | Job queued to run                                     |
  | `job.in_progress` | Job has started                                       |
  | `job.completed`   | Job finished processing                               |
  | `job.stopped`     | Job has stopped                                       |
  | `job.failed`      | Job failed to initialize without processing any files |
</ParamField>

<ParamField type="string">
  Channel description. Maximum 255 characters.
</ParamField>

<ParamField type="boolean">
  Whether the channel is active. Default: `true`.
</ParamField>

<ParamField type="string">
  Webhook endpoint URL. Must use HTTPS. Required when `channel_type` is `webhook`.
</ParamField>

<ParamField type="string">
  Signing secret for verifying incoming webhook requests. Must be  between24 and 75 bytes (24 to 75 ASCII characters). Applies when `channel_type` is `webhook`.

  Use this secret to [verify incoming webhook requests](/api-reference/webhooks#verify-webhook-requests). If you do not provide a secret, you cannot use secret-based verification for incoming webhook requests.

  <Note>
    The secret is write-only and is not returned by the API after creation. To change it later, use the [update channel endpoint](/api-reference/api/notification/update-notification-channel) instead.
  </Note>
</ParamField>

<ParamField type="object">
  Email delivery configuration. Required when `channel_type` is `email`.

  | Value             | Required | Description                                                                  |
  | ----------------- | -------- | ---------------------------------------------------------------------------- |
  | `recipient_email` | yes      | Email address to deliver notifications to.                                   |
  | `reply_to`        |          | Email address set as the reply-to header in notification emails.             |
  | `cc`              |          | Additional email addresses to copy on notification emails. Array of strings. |
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the channel.
</ResponseField>

<ResponseField name="channel_type" type="string">
  Channel type: `webhook` or `email`.
</ResponseField>

<ResponseField name="event_types" type="array">
  Subscribed event types.
</ResponseField>

<ResponseField name="enabled" type="boolean">
  Whether the channel is active.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the channel was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the channel was last updated.
</ResponseField>

<ResponseField name="description" type="string">
  Channel description.
</ResponseField>

<ResponseField name="url" type="string">
  Webhook endpoint URL. Present when `channel_type` is `webhook`.
</ResponseField>

<ResponseField name="email_config" type="object">
  Email delivery configuration. Present when `channel_type` is `email`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "channel_type": "webhook",
      "url": "https://hooks.example.com/notify",
      "event_types": ["job.completed", "job.failed"],
      "description": "Job status alerts",
      "enabled": true
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateChannelApiV1NotificationsChannelsPostRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.create_channel(
      request=CreateChannelApiV1NotificationsChannelsPostRequest(
          channel_type="webhook",
          url="https://hooks.example.com/notify",
          event_types=["job.completed", "job.failed"],
          description="Job status alerts",
          enabled=True,
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateChannelApiV1NotificationsChannelsPostRequest

  async def create_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.create_channel_async(
          request=CreateChannelApiV1NotificationsChannelsPostRequest(
              channel_type="webhook",
              url="https://hooks.example.com/notify",
              event_types=["job.completed", "job.failed"],
              description="Job status alerts",
              enabled=True,
          )
      )
      print(response)

  asyncio.run(create_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
    "channel_type": "webhook",
    "description": "Job status alerts",
    "event_types": ["job.completed", "job.failed"],
    "enabled": true,
    "url": "https://hooks.example.com/notify",
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T10:00:00Z"
  }
  ```
</ResponseExample>


# Delete notification channel
Source: https://docs.unstructured.io/api-reference/api/notification/delete-notification-channel

DELETE /api/v1/notifications/channels/{channel_id}
Delete workspace-scoped notification channel.

<Warning>
  Deleting a notification channel is a permanent action and is not recoverable.
</Warning>

## Path parameters

<ParamField type="string">
  The unique identifier of the notification channel to delete.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request DELETE \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteChannelApiV1NotificationsChannelsChannelIdDeleteRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.delete_channel(
      request=DeleteChannelApiV1NotificationsChannelsChannelIdDeleteRequest(
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteChannelApiV1NotificationsChannelsChannelIdDeleteRequest

  async def delete_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.delete_channel_async(
          request=DeleteChannelApiV1NotificationsChannelsChannelIdDeleteRequest(
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          )
      )
      print(response)

  asyncio.run(delete_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Get notification
Source: https://docs.unstructured.io/api-reference/api/notification/get-notification

GET /api/v1/notifications/{notification_id}
Get a single notification event by ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the notification event.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the notification event.
</ResponseField>

<ResponseField name="event_type" type="string">
  Event type (e.g., `job.completed`).
</ResponseField>

<ResponseField name="payload" type="object">
  Event payload data.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the event was created.
</ResponseField>

<ResponseField name="workflow_id" type="string">
  Workflow associated with this event. `null` for workspace-level events.
</ResponseField>

<ResponseField name="is_read" type="boolean">
  Whether the current user has marked this notification as read. Default: `false`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetNotificationApiV1NotificationsNotificationIdGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.get_notification(
      request=GetNotificationApiV1NotificationsNotificationIdGetRequest(
          notification_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetNotificationApiV1NotificationsNotificationIdGetRequest

  async def get_notification():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.get_notification_async(
          request=GetNotificationApiV1NotificationsNotificationIdGetRequest(
              notification_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
          )
      )
      print(response)

  asyncio.run(get_notification())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "event_type": "job.completed",
    "workflow_id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
    "payload": {
      "job_id": "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
      "status": "COMPLETED"
    },
    "created_at": "2026-04-29T10:05:00Z",
    "is_read": false
  }
  ```
</ResponseExample>


# Get notification channel
Source: https://docs.unstructured.io/api-reference/api/notification/get-notification-channel

GET /api/v1/notifications/channels/{channel_id}
Get workspace-scoped notification channel by ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the notification channel.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the channel.
</ResponseField>

<ResponseField name="channel_type" type="string">
  Channel type: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ResponseField>

<ResponseField name="event_types" type="array">
  Subscribed event types.
</ResponseField>

<ResponseField name="enabled" type="boolean">
  Whether the channel is active.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the channel was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the channel was last updated.
</ResponseField>

<ResponseField name="description" type="string">
  Channel description.
</ResponseField>

<ResponseField name="url" type="string">
  Webhook endpoint URL. Present when `channel_type` is `webhook`.
</ResponseField>

<ResponseField name="email_config" type="object">
  Email delivery configuration. Present when `channel_type` is `email`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetChannelApiV1NotificationsChannelsChannelIdGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.get_channel(
      request=GetChannelApiV1NotificationsChannelsChannelIdGetRequest(
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetChannelApiV1NotificationsChannelsChannelIdGetRequest

  async def get_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.get_channel_async(
          request=GetChannelApiV1NotificationsChannelsChannelIdGetRequest(
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          )
      )
      print(response)

  asyncio.run(get_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
    "channel_type": "webhook",
    "description": "Job status alerts",
    "event_types": ["job.completed", "job.failed"],
    "enabled": true,
    "url": "https://hooks.example.com/notify",
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T10:00:00Z"
  }
  ```
</ResponseExample>


# Get notifications unread count
Source: https://docs.unstructured.io/api-reference/api/notification/get-notifications-unread-count

GET /api/v1/notifications/unread-count
Get count of unread notification events for the current user.

## Query parameters

<ParamField type="string">
  Filter unread count to a specific workflow.
</ParamField>

## Response

<ResponseField name="unread_count" type="integer">
  Number of unread notification events.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/unread-count" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetUnreadCountApiV1NotificationsUnreadCountGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.get_unread_count(
      request=GetUnreadCountApiV1NotificationsUnreadCountGetRequest()
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetUnreadCountApiV1NotificationsUnreadCountGetRequest

  async def get_unread_count():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.get_unread_count_async(
          request=GetUnreadCountApiV1NotificationsUnreadCountGetRequest()
      )
      print(response)

  asyncio.run(get_unread_count())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "unread_count": 5
  }
  ```
</ResponseExample>


# List notification channels
Source: https://docs.unstructured.io/api-reference/api/notification/list-notification-channels

GET /api/v1/notifications/channels
List workspace-scoped notification channels.

## Query parameters

<ParamField type="string">
  Channel type to filter by: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ParamField>

<ParamField type="boolean">
  Filter by enabled status.
</ParamField>

## Response

<ResponseField name="items" type="array">
  List of notification channel objects.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels?channel_type=webhook" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListChannelsApiV1NotificationsChannelsGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.list_channels(
      request=ListChannelsApiV1NotificationsChannelsGetRequest(
          channel_type="webhook",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListChannelsApiV1NotificationsChannelsGetRequest

  async def list_channels():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.list_channels_async(
          request=ListChannelsApiV1NotificationsChannelsGetRequest(
              channel_type="webhook",
          )
      )
      print(response)

  asyncio.run(list_channels())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "items": [
      {
        "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
        "channel_type": "webhook",
        "description": "Job status alerts",
        "event_types": ["job.completed", "job.failed"],
        "enabled": true,
        "url": "https://hooks.example.com/notify",
        "created_at": "2026-04-29T10:00:00Z",
        "updated_at": "2026-04-29T10:00:00Z"
      }
    ]
  }
  ```
</ResponseExample>


# List notifications
Source: https://docs.unstructured.io/api-reference/api/notification/list-notifications

GET /api/v1/notifications
List notification events for the authenticated user, ordered by creation date with cursor-based pagination.

## Query parameters

<ParamField type="string">
  Filter by workflow ID.
</ParamField>

<ParamField type="string">
  Comma-separated list of event types to filter by (e.g., `job.completed,job.failed`).
</ParamField>

<ParamField type="string">
  Return events created after this ISO 8601 timestamp.
</ParamField>

<ParamField type="integer">
  Maximum number of events to return. Must be between 1 and 100. Default: `50`.
</ParamField>

<ParamField type="string">
  Pagination cursor from a previous response.
</ParamField>

<ParamField type="boolean">
  If `true`, return only unread events for the current user. Default: `false`.
</ParamField>

## Response

<ResponseField name="events" type="array">
  List of notification event objects.
</ResponseField>

<ResponseField name="next_cursor" type="string">
  Cursor for the next page of results. `null` if no further pages exist.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications?limit=50&unread_only=false" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListNotificationsApiV1NotificationsGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.list_notifications(
      request=ListNotificationsApiV1NotificationsGetRequest(
          limit=50,
          unread_only=False,
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListNotificationsApiV1NotificationsGetRequest

  async def list_notifications():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.list_notifications_async(
          request=ListNotificationsApiV1NotificationsGetRequest(
              limit=50,
              unread_only=False,
          )
      )
      print(response)

  asyncio.run(list_notifications())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "events": [
      {
        "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
        "event_type": "job.completed",
        "workflow_id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
        "payload": {
          "job_id": "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
          "status": "COMPLETED"
        },
        "created_at": "2026-04-29T10:05:00Z",
        "is_read": false
      }
    ],
    "next_cursor": null
  }
  ```
</ResponseExample>


# Mark notifications read
Source: https://docs.unstructured.io/api-reference/api/notification/mark-notifications-read

POST /api/v1/notifications/mark-read
Mark notification events as read for the current user.

Provide exactly one of `notification_ids`, `before`, or `mark_all`. `workflow_id` is only valid when used with `before` or `mark_all`.

This results in the following behaviors:

* If you specify `notification_ids` only, it will mark all notifications with the specified IDs as read.
* If you specify `before` only, it will mark all notifications created in the workspace before the specified timestamp as read.
* If you specify `mark_all` only, it will mark at the time of the API call all existing notifications in the workspace as read.
* If you specify `workflow_id` and `before`, it will mark all notifications created before the specified timestamp for the specified workflow as read.
* If you specify `workflow_id` and `mark_all`, it will mark at the time of the API call all existing notifications associated with the specified workflow as read.

## Body

<ParamField type="array">
  Specific notification IDs to mark as read. Maximum 100 UUIDs.
</ParamField>

<ParamField type="string">
  Mark all notifications created before this ISO 8601 timestamp (in format `YYYY-MM-DDTHH:MM:SSZ`) as read.
</ParamField>

<ParamField type="boolean">
  If `true`, mark all unread notifications as read. Default: `false`.
</ParamField>

<ParamField type="string">
  Scope the operation to a specific workflow. Only valid when used with `before` or `mark_all`.
</ParamField>

## Response

<ResponseField name="marked_count" type="integer">
  Number of notifications newly marked as read.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/mark-read" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "notification_ids": [
        "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
        "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e"
      ]
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import MarkReadApiV1NotificationsMarkReadPostRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.mark_read(
      request=MarkReadApiV1NotificationsMarkReadPostRequest(
          notification_ids=[
              "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
              "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
          ],
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import MarkReadApiV1NotificationsMarkReadPostRequest

  async def mark_read():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.mark_read_async(
          request=MarkReadApiV1NotificationsMarkReadPostRequest(
              notification_ids=[
                  "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
                  "b2c3d4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e",
              ],
          )
      )
      print(response)

  asyncio.run(mark_read())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "marked_count": 2
  }
  ```
</ResponseExample>


# Notification endpoints
Source: https://docs.unstructured.io/api-reference/api/notification/notification-apis

Read and manage workspace-level notifications and the delivery channels that route them. Configure email or webhook targets, verify them, and track unread activity across your workflows.

For more information about notification channels, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).

| Name                                                                                             | Method   | Path                                                 |
| ------------------------------------------------------------------------------------------------ | -------- | ---------------------------------------------------- |
| [List notifications](/api-reference/api/notification/list-notifications)                         | `GET`    | `/api/v1/notifications`                              |
| [Get notification](/api-reference/api/notification/get-notification)                             | `GET`    | `/api/v1/notifications/{notification_id}`            |
| [Mark notifications read](/api-reference/api/notification/mark-notifications-read)               | `POST`   | `/api/v1/notifications/mark-read`                    |
| [Get notifications unread count](/api-reference/api/notification/get-notifications-unread-count) | `GET`    | `/api/v1/notifications/unread-count`                 |
| [List notification channels](/api-reference/api/notification/list-notification-channels)         | `GET`    | `/api/v1/notifications/channels`                     |
| [Get notification channel](/api-reference/api/notification/get-notification-channel)             | `GET`    | `/api/v1/notifications/channels/{channel_id}`        |
| [Create notification channel](/api-reference/api/notification/create-notification-channel)       | `POST`   | `/api/v1/notifications/channels`                     |
| [Update notification channel](/api-reference/api/notification/update-notification-channel)       | `PATCH`  | `/api/v1/notifications/channels/{channel_id}`        |
| [Delete notification channel](/api-reference/api/notification/delete-notification-channel)       | `DELETE` | `/api/v1/notifications/channels/{channel_id}`        |
| [Verify notification channel](/api-reference/api/notification/verify-notification-channel)       | `POST`   | `/api/v1/notifications/channels/{channel_id}/verify` |

To read and manage *workflow-scoped* notification channels, use the following endpoints:

* [List workflow notification channels](/api-reference/api/workflow/list-workflow-notification-channels)
* [Get workflow notification channel](/api-reference/api/workflow/get-workflow-notification-channel)
* [Create workflow notification channel](/api-reference/api/workflow/create-workflow-notification-channel)
* [Update workflow notification channel](/api-reference/api/workflow/update-workflow-notification-channel)
* [Delete workflow notification channel](/api-reference/api/workflow/delete-workflow-notification-channel)
* [Verify workflow notification channel](/api-reference/api/workflow/verify-workflow-notification-channel)


# Update notification channel
Source: https://docs.unstructured.io/api-reference/api/notification/update-notification-channel

PATCH /api/v1/notifications/channels/{channel_id}
Update workspace-scoped notification channel. Only provided fields are updated.

Channels that have not been [verified](/api-reference/api/notification/verify-notification-channel) cannot be updated.

## Path parameters

<ParamField type="string">
  The unique identifier of the notification channel to update.
</ParamField>

## Body

<ParamField type="string">
  Updated channel description. Maximum 255 characters.
</ParamField>

<ParamField type="array">
  Updated list of event types to subscribe to.

  | Value             | Description                                           |
  | ----------------- | ----------------------------------------------------- |
  | `job.scheduled`   | Job queued to run                                     |
  | `job.in_progress` | Job has started                                       |
  | `job.completed`   | Job finished processing                               |
  | `job.stopped`     | Job has stopped                                       |
  | `job.failed`      | Job failed to initialize without processing any files |
</ParamField>

<ParamField type="boolean">
  Whether the channel is active.
</ParamField>

<ParamField type="string">
  Updated webhook endpoint URL. Applies when `channel_type` is `webhook`.
</ParamField>

<ParamField type="string">
  Updated signing secret for verifying incoming webhook requests. Must be  between 24 and 75 bytes (24 to 75 ASCII characters). Applies when `channel_type` is `webhook`. Omit to keep the current value.

  Use this secret to [verify incoming webhook requests](/api-reference/webhooks#verify-webhook-requests). If you do not provide a secret, you cannot use secret-based verification for incoming webhook requests.
</ParamField>

<ParamField type="object">
  Updated email delivery configuration. Applies when `channel_type` is `email`.

  | Value             | Required | Description                                                                  |
  | ----------------- | -------- | ---------------------------------------------------------------------------- |
  | `recipient_email` | yes      | Email address to deliver notifications to.                                   |
  | `reply_to`        |          | Email address set as the reply-to header in notification emails.             |
  | `cc`              |          | Additional email addresses to copy on notification emails. Array of strings. |
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the channel.
</ResponseField>

<ResponseField name="channel_type" type="string">
  Channel type: `webhook` or `email`.
</ResponseField>

<ResponseField name="event_types" type="array">
  Subscribed event types.
</ResponseField>

<ResponseField name="enabled" type="boolean">
  Whether the channel is active.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the channel was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the channel was last updated.
</ResponseField>

<ResponseField name="description" type="string">
  Channel description.
</ResponseField>

<ResponseField name="url" type="string">
  Webhook endpoint URL. Present when `channel_type` is `webhook`.
</ResponseField>

<ResponseField name="email_config" type="object">
  Email delivery configuration. Present when `channel_type` is `email`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request PATCH \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "event_types": ["job.completed", "job.failed", "job.in_progress"],
      "enabled": true
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateChannelApiV1NotificationsChannelsChannelIdPatchRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.update_channel(
      request=UpdateChannelApiV1NotificationsChannelsChannelIdPatchRequest(
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          event_types=["job.completed", "job.failed", "job.in_progress"],
          enabled=True,
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateChannelApiV1NotificationsChannelsChannelIdPatchRequest

  async def update_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.update_channel_async(
          request=UpdateChannelApiV1NotificationsChannelsChannelIdPatchRequest(
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
              event_types=["job.completed", "job.failed", "job.in_progress"],
              enabled=True,
          )
      )
      print(response)

  asyncio.run(update_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
    "channel_type": "webhook",
    "description": "Job status alerts",
    "event_types": ["job.completed", "job.failed", "job.in_progress"],
    "enabled": true,
    "url": "https://hooks.example.com/notify",
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T11:00:00Z"
  }
  ```
</ResponseExample>


# Verify notification channel
Source: https://docs.unstructured.io/api-reference/api/notification/verify-notification-channel

POST /api/v1/notifications/channels/{channel_id}/verify
Verify a notification channel using a verification code.

<Note>
  Email channels require verification. A 6-digit verification code is sent to the recipient email address when the channel is created. Webhook channels do not require verification.

  For more information, see [Verify the email channel](/api-reference/email#verify-the-email-channel).
</Note>

## Path parameters

<ParamField type="string">
  The unique identifier of the notification channel to verify.
</ParamField>

## Body

<ParamField type="string">
  The 6-digit verification code sent to the recipient email address during channel creation.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f/verify" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "verification_code": "123456"
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import VerifyChannelApiV1NotificationsChannelsChannelIdVerifyPostRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.notifications.verify_channel(
      request=VerifyChannelApiV1NotificationsChannelsChannelIdVerifyPostRequest(
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          verification_code="123456",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import VerifyChannelApiV1NotificationsChannelsChannelIdVerifyPostRequest

  async def verify_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.notifications.verify_channel_async(
          request=VerifyChannelApiV1NotificationsChannelsChannelIdVerifyPostRequest(
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
              verification_code="123456",
          )
      )
      print(response)

  asyncio.run(verify_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Create source connector
Source: https://docs.unstructured.io/api-reference/api/source/create-source

POST /api/v1/sources/
Create a new source connector using the provided configuration.

## Body

<ParamField type="string">
  Source connector name.
</ParamField>

<ParamField type="string">
  Connector type. One of: `azure`, `box`, `confluence`, `couchbase`, `databricks_volumes`, `dropbox`, `elasticsearch`, `gcs`, `google_drive`, `kafka-cloud`, `mongodb`, `onedrive`, `opensearch`, `outlook`, `postgres`, `s3`, `salesforce`, `sharepoint`, `snowflake`, `teradata`, `jira`, `zendesk`.
</ParamField>

<ParamField type="object">
  Connector configuration. Required fields vary by connector `type`.

  For the specific settings to include for a connector, see the entry for that connector in [Sources](/api-reference/workflow/sources/overview).
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the source connector.
</ResponseField>

<ResponseField name="name" type="string">
  Source connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.

  For the specific settings to include for a specific connector, see the entry for that connector in [Sources](/api-reference/workflow/sources/overview).
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "name": "My S3 Source",
      "type": "s3",
      "config": {
        "remote_url": "s3://my-bucket/input/",
        "access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "recursive": true
      }
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.create_source(
      request=CreateSourceRequest(
          create_source_connector=CreateSourceConnector(
              name="My S3 Source",
              type="s3",
              config={
                  "remote_url": "s3://my-bucket/input/",
                  "access_key_id": "AKIAIOSFODNN7EXAMPLE",
                  "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
                  "recursive": True,
              },
          )
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  async def create_source():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.create_source_async(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="My S3 Source",
                  type="s3",
                  config={
                      "remote_url": "s3://my-bucket/input/",
                      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
                      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
                      "recursive": True,
                  },
              )
          )
      )
      print(response)

  asyncio.run(create_source())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "name": "My S3 Source",
    "type": "s3",
    "config": {
      "remote_url": "s3://my-bucket/input/",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
      "recursive": true
    },
    "created_at": "2026-01-01T00:00:00Z",
    "updated_at": null
  }
  ```
</ResponseExample>


# Create source connection check
Source: https://docs.unstructured.io/api-reference/api/source/create-source-connection-check

POST /api/v1/sources/{source_id}/connection-check
Test a source connection by creating a connection check for the specified source connector.

## Path parameters

<ParamField type="string">
  The unique identifier of the source connector.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the connection check.
</ResponseField>

<ResponseField name="status" type="string">
  Connection check status. One of: `SCHEDULED`, `SUCCESS`, `FAILURE`.
</ResponseField>

<ResponseField name="reason" type="string">
  Failure reason, if applicable.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the check was initiated.
</ResponseField>

<ResponseField name="reported_at" type="string">
  ISO 8601 timestamp when the check result was reported.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/connection-check" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateConnectionCheckSourcesRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.create_source_connection_check(
      request=CreateConnectionCheckSourcesRequest(
          source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateConnectionCheckSourcesRequest

  async def create_source_connection_check():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.create_source_connection_check_async(
          request=CreateConnectionCheckSourcesRequest(
              source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
          )
      )
      print(response)

  asyncio.run(create_source_connection_check())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c2b3a4d5-6e7f-8a9b-0c1d-2e3f4a5b6c7d",
    "status": "SCHEDULED",
    "reason": null,
    "created_at": "2026-01-01T00:00:00Z",
    "reported_at": null
  }
  ```
</ResponseExample>


# Delete source connector
Source: https://docs.unstructured.io/api-reference/api/source/delete-source

DELETE /api/v1/sources/{source_id}
Delete a specific source connector identified by its ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the source connector.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request DELETE \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteSourceRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.delete_source(
      request=DeleteSourceRequest(
          source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteSourceRequest

  async def delete_source():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.delete_source_async(
          request=DeleteSourceRequest(
              source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
          )
      )
      print(response)

  asyncio.run(delete_source())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Get source connector
Source: https://docs.unstructured.io/api-reference/api/source/get-source

GET /api/v1/sources/{source_id}
Retrieve detailed information for a specific source connector.

## Path parameters

<ParamField type="string">
  The unique identifier of the source connector.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the source connector.
</ResponseField>

<ResponseField name="name" type="string">
  Source connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetSourceRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.get_source(
      request=GetSourceRequest(
          source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetSourceRequest

  async def get_source():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.get_source_async(
          request=GetSourceRequest(
              source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
          )
      )
      print(response)

  asyncio.run(get_source())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "name": "My S3 Source",
    "type": "s3",
    "config": {
      "remote_url": "s3://my-bucket/input/",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
      "recursive": true
    },
    "created_at": "2026-01-01T00:00:00Z",
    "updated_at": "2026-01-15T12:00:00Z"
  }
  ```
</ResponseExample>


# Get source connection check
Source: https://docs.unstructured.io/api-reference/api/source/get-source-connection-check

GET /api/v1/sources/{source_id}/connection-check
Retrieve the most recent connection check for the source connector.

## Path parameters

<ParamField type="string">
  The unique identifier of the source connector.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the connection check.
</ResponseField>

<ResponseField name="status" type="string">
  Connection check status. One of: `SCHEDULED`, `SUCCESS`, `FAILURE`.
</ResponseField>

<ResponseField name="reason" type="string">
  Failure reason, if applicable.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the check was initiated.
</ResponseField>

<ResponseField name="reported_at" type="string">
  ISO 8601 timestamp when the check result was reported.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d/connection-check" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetConnectionCheckSourcesRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.get_source_connection_check(
      request=GetConnectionCheckSourcesRequest(
          source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetConnectionCheckSourcesRequest

  async def get_source_connection_check():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.get_source_connection_check_async(
          request=GetConnectionCheckSourcesRequest(
              source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d"
          )
      )
      print(response)

  asyncio.run(get_source_connection_check())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c2b3a4d5-6e7f-8a9b-0c1d-2e3f4a5b6c7d",
    "status": "SUCCESS",
    "reason": null,
    "created_at": "2026-01-01T00:00:00Z",
    "reported_at": "2026-01-01T00:00:05Z"
  }
  ```
</ResponseExample>


# List source connectors
Source: https://docs.unstructured.io/api-reference/api/source/list-sources

GET /api/v1/sources/
Retrieve a list of available source connectors.

## Query parameters

<ParamField type="string">
  Filter by connector type. One of: `azure`, `box`, `confluence`, `couchbase`, `databricks_volumes`, `dropbox`, `elasticsearch`, `gcs`, `google_drive`, `kafka-cloud`, `mongodb`, `onedrive`, `opensearch`, `outlook`, `postgres`, `s3`, `salesforce`, `sharepoint`, `snowflake`, `teradata`, `jira`, `zendesk`.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the source connector.
</ResponseField>

<ResponseField name="name" type="string">
  Source connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListSourcesRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.list_sources(
      request=ListSourcesRequest()
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListSourcesRequest

  async def list_sources():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.list_sources_async(
          request=ListSourcesRequest()
      )
      print(response)

  asyncio.run(list_sources())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  [
    {
      "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      "name": "My S3 Source",
      "type": "s3",
      "config": {
        "remote_url": "s3://my-bucket/input/",
        "access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "recursive": true
      },
      "created_at": "2026-01-01T00:00:00Z",
      "updated_at": "2026-01-15T12:00:00Z"
    }
  ]
  ```
</ResponseExample>


# Source endpoints
Source: https://docs.unstructured.io/api-reference/api/source/source-apis



Manage the source connectors that feed documents into your Unstructured workflows. Create and configure connections to storage systems, verify connectivity, and keep connectors up to date.

| Name                                                                                       | Method   | Path                                           |
| ------------------------------------------------------------------------------------------ | -------- | ---------------------------------------------- |
| [List source connectors](/api-reference/api/source/list-sources)                           | `GET`    | `/api/v1/sources/`                             |
| [Get source connector](/api-reference/api/source/get-source)                               | `GET`    | `/api/v1/sources/{source_id}`                  |
| [Create source connector](/api-reference/api/source/create-source)                         | `POST`   | `/api/v1/sources/`                             |
| [Update source connector](/api-reference/api/source/update-source)                         | `PUT`    | `/api/v1/sources/{source_id}`                  |
| [Delete source connector](/api-reference/api/source/delete-source)                         | `DELETE` | `/api/v1/sources/{source_id}`                  |
| [Get source connection check](/api-reference/api/source/get-source-connection-check)       | `GET`    | `/api/v1/sources/{source_id}/connection-check` |
| [Create source connection check](/api-reference/api/source/create-source-connection-check) | `POST`   | `/api/v1/sources/{source_id}/connection-check` |


# Update source connector
Source: https://docs.unstructured.io/api-reference/api/source/update-source

PUT /api/v1/sources/{source_id}
Update the configuration of an existing source connector.

You must specify all of the settings for the connector, even for settings that are not changing.

You can change any of the connector's settings except for its `name`and `type`.

## Path parameters

<ParamField type="string">
  The unique identifier of the source connector.
</ParamField>

## Body

<ParamField type="string">
  New source connector name.
</ParamField>

<ParamField type="object">
  Updated connector configuration. Fields vary by connector type.

  For the specific settings to include for a connector, see the entry for that connector in [Sources](/api-reference/workflow/sources/overview).
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the source connector.
</ResponseField>

<ResponseField name="name" type="string">
  Source connector name.
</ResponseField>

<ResponseField name="type" type="string">
  Connector type.
</ResponseField>

<ResponseField name="config" type="object">
  Connector configuration. Fields vary by connector `type`.

  For the specific settings to include for a specific connector, see the entry for that connector in [Sources](/api-reference/workflow/sources/overview).
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the connector was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the connector was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request PUT \
    --url "${UNSTRUCTURED_API_URL}/api/v1/sources/a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "name": "My Updated S3 Source",
      "config": {
        "remote_url": "s3://my-new-bucket/input/",
        "access_key_id": "AKIAIOSFODNN7EXAMPLE",
        "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "recursive": true
      }
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateSourceRequest
  from unstructured_client.models.shared import UpdateSourceConnector

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.sources.update_source(
      request=UpdateSourceRequest(
          source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
          update_source_connector=UpdateSourceConnector(
              name="My Updated S3 Source",
              config={
                  "remote_url": "s3://my-new-bucket/input/",
                  "access_key_id": "AKIAIOSFODNN7EXAMPLE",
                  "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
                  "recursive": True,
              },
          )
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateSourceRequest
  from unstructured_client.models.shared import UpdateSourceConnector

  async def update_source():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.sources.update_source_async(
          request=UpdateSourceRequest(
              source_id="a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
              update_source_connector=UpdateSourceConnector(
                  name="My Updated S3 Source",
                  config={
                      "remote_url": "s3://my-new-bucket/input/",
                      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
                      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
                      "recursive": True,
                  },
              )
          )
      )
      print(response)

  asyncio.run(update_source())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "name": "My Updated S3 Source",
    "type": "s3",
    "config": {
      "remote_url": "s3://my-new-bucket/input/",
      "access_key_id": "AKIAIOSFODNN7EXAMPLE",
      "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
      "recursive": true
    },
    "created_at": "2026-01-01T00:00:00Z",
    "updated_at": "2026-04-29T10:00:00Z"
  }
  ```
</ResponseExample>


# Get template
Source: https://docs.unstructured.io/api-reference/api/template/get-template

GET /api/v1/templates/{template_id}
Retrieve detailed information and DAG for a specific template.

## Path parameters

<ParamField type="string">
  The unique identifier of the template.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the template.
</ResponseField>

<ResponseField name="name" type="string">
  Template name.
</ResponseField>

<ResponseField name="type" type="string">
  Template type category.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the template was created.
</ResponseField>

<ResponseField name="is_custom" type="boolean">
  Whether this is a user-created template (`true`) or a platform-provided template (`false`).
</ResponseField>

<ResponseField name="description" type="string">
  Template description.
</ResponseField>

<ResponseField name="tags" type="array">
  Classification labels for discovery.
</ResponseField>

<ResponseField name="dag" type="object">
  Full DAG configuration for the template, including node definitions and processing instructions.

  For more information, see [Workflow Nodes](/api-reference/workflow/nodes/overview).
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/templates/t1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetTemplateRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.templates.get_template(
      request=GetTemplateRequest(
          template_id="t1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetTemplateRequest

  async def get_template():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.templates.get_template_async(
          request=GetTemplateRequest(
              template_id="t1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c"
          )
      )
      print(response)

  asyncio.run(get_template())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "t1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
    "name": "Basic ETL",
    "description": "A simple extract, transform, and load workflow template.",
    "type": "etl",
    "created_at": "2026-01-01T00:00:00Z",
    "tags": ["etl", "basic"],
    "is_custom": false,
    "dag": {
      "nodes": [
        {
          "id": "n1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
          "node_type": "partition",
          "node_subtype": "auto",
          "config": {}
        }
      ]
    }
  }
  ```
</ResponseExample>


# List templates
Source: https://docs.unstructured.io/api-reference/api/template/list-templates

GET /api/v1/templates/
Retrieve a list of available templates with their metadata.

## Response

<ResponseField name="id" type="string">
  Unique identifier for the template.
</ResponseField>

<ResponseField name="name" type="string">
  Template name.
</ResponseField>

<ResponseField name="type" type="string">
  Template type category.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the template was created.
</ResponseField>

<ResponseField name="is_custom" type="boolean">
  Whether this is a user-created template (`true`) or a platform-provided template (`false`).
</ResponseField>

<ResponseField name="description" type="string">
  Template description.
</ResponseField>

<ResponseField name="tags" type="array">
  Classification labels for discovery.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/templates/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListTemplatesRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.templates.list_templates(
      request=ListTemplatesRequest()
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListTemplatesRequest

  async def list_templates():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.templates.list_templates_async(
          request=ListTemplatesRequest()
      )
      print(response)

  asyncio.run(list_templates())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  [
    {
      "id": "t1a2b3c4-5d6e-7f8a-9b0c-1d2e3f4a5b6c",
      "name": "Basic ETL",
      "description": "A simple extract, transform, and load workflow template.",
      "type": "etl",
      "created_at": "2026-01-01T00:00:00Z",
      "tags": ["etl", "basic"],
      "is_custom": false
    }
  ]
  ```
</ResponseExample>


# Template endpoints
Source: https://docs.unstructured.io/api-reference/api/template/template-apis



Browse the predefined workflow templates available in your account. Use them as a starting point when creating new workflows rather than building from scratch.

| Name                                                         | Method | Path                              |
| ------------------------------------------------------------ | ------ | --------------------------------- |
| [List templates](/api-reference/api/template/list-templates) | `GET`  | `/api/v1/templates/`              |
| [Get template](/api-reference/api/template/get-template)     | `GET`  | `/api/v1/templates/{template_id}` |


# Create workflow
Source: https://docs.unstructured.io/api-reference/api/workflow/create-workflow

POST /api/v1/workflows/
Create a new workflow, either custom or auto, and configure its settings.

<Note>
  This endpoint creates a workflow that persists until it is explicitly deleted (a *long-lived workflow*). To create a workflow that exists only for the duration of a single job run using local files as input, use the [create job endpoint](/api-reference/api/job/create-job) instead.
</Note>

## Body

<ParamField type="string">
  Workflow name.
</ParamField>

<ParamField type="string">
  Execution mode. `auto` uses sensible default workflow settings to enable you to get good-quality results faster. `custom` enables you to fine-tune the workflow settings to get very specific results.
</ParamField>

<Note>
  The workflow types `advanced`, `basic`, and `platinum` are non-operational and will be removed in a future release.
</Note>

<Note>
  Workflows with no `source_id` use a local file source. Local-source workflows must set `workflow_type` to `custom`, cannot be set to run on a repeating schedule, and cannot be run from Unstructured Pipelines (though they can be run via the API or Python SDK).
</Note>

<ParamField type="string">
  ID of the source connector.
</ParamField>

<ParamField type="string">
  ID of the destination connector.
</ParamField>

<ParamField type="array">
  Processing pipeline stages. Each node requires `id` (string, UUID) and `node_type` (string), and supports optional `node_subtype` (string), `config` (object), and `params` (object).

  For more information on workflow nodes, see [Workflow nodes](/api-reference/workflow/nodes/overview).
</ParamField>

<ParamField type="string">
  ID of a pre-built workflow template to use as the basis for the workflow.
</ParamField>

<ParamField type="string">
  Repeating run schedule. Valid values and their cron equivalents:

  | Value              | cron           | Description                                          |
  | ------------------ | -------------- | ---------------------------------------------------- |
  | `every 15 minutes` | `*/15 * * * *` | Every 15 minutes.                                    |
  | `every hour`       | `0 * * * *`    | At the first minute of every hour.                   |
  | `every 2 hours`    | `0 */2 * * *`  | At the first minute of every second hour.            |
  | `every 4 hours`    | `0 */4 * * *`  | At the first minute of every fourth hour.            |
  | `every 6 hours`    | `0 */6 * * *`  | At the first minute of every sixth hour.             |
  | `every 8 hours`    | `0 */8 * * *`  | At the first minute of every eighth hour.            |
  | `every 10 hours`   | `0 */10 * * *` | At the first minute of every tenth hour.             |
  | `every 12 hours`   | `0 */12 * * *` | At the first minute of every twelfth hour.           |
  | `daily`            | `0 0 * * *`    | At the first minute of every day.                    |
  | `weekly`           | `0 0 * * 0`    | At the first minute of every Sunday.                 |
  | `monthly`          | `0 0 1 * *`    | At the first minute of the first day of every month. |

  If omitted, the workflow does not automatically run on a repeating schedule.

  Workflows with a local source cannot be set to run on a repeating schedule.
</ParamField>

<ParamField type="boolean">
  Default: `false`. If `true`, reprocesses all documents in the source location on every run. If `false`, the workflow excludes from future processing any files Unstructured determines are unchanged since the last time the workflow ran.

  Unstructured determines if a document has changed based on the document version. For each workflow, Unstructured maintains a record of documents (and their versions, if present) processed by that workflow. Each document record consists of:

  * A `record_id` derived from the document name and path.
  * A `record_version` derived from either the document Etag (if the source provider generates one) or the source provider's native version identifier.

  When you set `reprocess_all` to `false` for a source connector that supports `reprocess_all`, Unstructured uses this list of records to determine whether or not to process each document:

  * If the `record_id` does not exist in the workflow records, Unstructured processes the document.
  * If the `record_id` exists, but the `record_version` has changed, or there is no `record_version`, Unstructured processes the document.

  The following table lists out the possible `record-id` and `record_version` combinations, and the action Unstructured takes in each case:

  | `record_id` | `record_version` | Action              |
  | ----------- | ---------------- | ------------------- |
  | Exists      | Unchanged        | Do not process file |
  | Exists      | Changed          | Process file        |
  | Exists      | (none)           | Process file        |
  | New         | (Does not apply) | Process file        |

  <Note>
    Renaming a document results in a new `record_id`; Unstructured will then reprocess the renamed document when the workflow runs.
  </Note>

  The following table lists the source connectors that support the `reprocess_all` setting. The **Record version base** column specifies the versioning information Unstructured uses to generated the corresponding record version for each processed document.

  Source connectors that do not support `reprocess_all` reprocess every document in the source location each time the workflow runs.

  | Connector                                                                | `record_version` base |
  | ------------------------------------------------------------------------ | --------------------- |
  | [Amazon S3](/api-reference/workflow/sources/s3)                          | ETag                  |
  | [Azure Blob Storage](/api-reference/workflow/sources/azure-blob-storage) | ETag                  |
  | [Box](/api-reference/workflow/sources/box)                               | Provider version ID   |
  | [Dropbox](/api-reference/workflow/sources/dropbox)                       | Provider version ID   |
  | [Elastisearch](/api-reference/workflow/sources/elasticsearch)            | Provider version ID   |
  | [Google Cloud Storage](/api-reference/workflow/sources/google-cloud)     | ETag                  |
  | [Google Drive](/api-reference/workflow/sources/google-drive)             | Provider version ID   |
  | [Microsoft OneDrive](/api-reference/workflow/sources/onedrive)           | Provider version ID   |
  | [Microsoft SharePoint](/api-reference/workflow/sources/sharepoint)       | Provider version ID   |

  Additional considerations to take into account when setting `reprocess_all` to `false`:

  * Unstructured only adds document records for documents that it successfully processes. Documents that failed to process will be reprocessed the next time the workflow is run.
  * Because S3 ETags are content-based, changing the metadata on an S3 object will not result in it being reprocessed.
  * For source providers that support the S3 protocol, be aware that deleting an object and then reuploading it to the source location will maintain the same `record_id`, but may result in a different `record_version` being generated. This is especially true of multipart uploads. This results in Unstructured reprocessing the document.
  * For source providers that offer Key Management Services (KMS), be aware that server-side encryption can change document ETags. This results in the the `record_version` of a document changing, and Unstructured reprocessing the document.
  * If you clone or recreate a source connector, the resulting connector does not include the document processing history of the previous connector.
  * Changing a workflow's configuration does not automatically result in Unstructured reprocessing all documents. For example, changing chunker, embedder, enrichment, or partitioner settings may not result in reprocessing all document. To reprocess all documents using new workflow settings, set `reprocess_all` to `true` for at least the next workflow run.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the workflow.
</ResponseField>

<ResponseField name="name" type="string">
  Workflow name.
</ResponseField>

<ResponseField name="workflow_type" type="string">
  Workflow type: `custom` or `auto`.
</ResponseField>

<ResponseField name="status" type="string">
  Workflow state: `active`, `inactive`, or `paused`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the workflow was created.
</ResponseField>

<ResponseField name="source_id" type="string">
  Source connector ID.
</ResponseField>

<ResponseField name="destination_id" type="string">
  Destination connector ID.
</ResponseField>

<ResponseField name="schedule" type="string">
  Repeating run schedule.
</ResponseField>

<ResponseField name="dag_nodes" type="array">
  Workflow processing pipeline nodes.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the workflow was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "name": "my-workflow",
      "workflow_type": "auto",
      "source_id": "7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
      "destination_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      "schedule": "daily"
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateWorkflowRequest
  from unstructured_client.models.shared import CreateWorkflow, WorkflowType

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.create_workflow(
      request=CreateWorkflowRequest(
          create_workflow=CreateWorkflow(
              name="my-workflow",
              workflow_type=WorkflowType.AUTO,
              source_id="7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
              destination_id="1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
              schedule="daily",
          )
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateWorkflowRequest
  from unstructured_client.models.shared import CreateWorkflow, WorkflowType

  async def create_workflow():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.create_workflow_async(
          request=CreateWorkflowRequest(
              create_workflow=CreateWorkflow(
                  name="my-workflow",
                  workflow_type=WorkflowType.AUTO,
                  source_id="7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
                  destination_id="1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
                  schedule="daily",
              )
          )
      )
      print(response)

  asyncio.run(create_workflow())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "9b8c7d6e-5f4a-3b2c-1d0e-9f8a7b6c5d4e",
    "name": "my-workflow",
    "workflow_type": "auto",
    "status": "active",
    "source_id": "7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
    "destination_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "schedule": "daily",
    "dag_nodes": null,
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": null
  }
  ```
</ResponseExample>


# Create workflow notification channel
Source: https://docs.unstructured.io/api-reference/api/workflow/create-workflow-notification-channel

POST /api/v1/workflows/{workflow_id}/notifications/channels
Create a new delivery channel for receiving platform event notifications scoped to the specified workflow.

<Warning>
  The `secret` field is write-only and is not returned by the API after creation. To change it later, use the [update workflow notification channel endpoint](/api-reference/api/workflow/update-workflow-notification-channel) instead.
</Warning>

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

## Body

<ParamField type="string">
  Channel type: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ParamField>

<ParamField type="array">
  Event types to subscribe to.

  | Value             | Description                                           |
  | ----------------- | ----------------------------------------------------- |
  | `job.scheduled`   | Job queued to run                                     |
  | `job.in_progress` | Job has started                                       |
  | `job.completed`   | Job finished processing                               |
  | `job.stopped`     | Job has stopped                                       |
  | `job.failed`      | Job failed to initialize without processing any files |
</ParamField>

<ParamField type="string">
  Channel description. Maximum 255 characters.
</ParamField>

<ParamField type="boolean">
  Whether the channel is active. Default: `true`.
</ParamField>

<ParamField type="string">
  Webhook endpoint URL. Must use HTTPS. Required when `channel_type` is `webhook`.
</ParamField>

<ParamField type="string">
  Signing secret for verifying incoming webhook requests. Must be between 24 and 75 bytes (24 to 75 ASCII characters). Applies when `channel_type` is `webhook`.

  Use this secret to [verify incoming webhook requests](/api-reference/webhooks#verify-webhook-requests). If you do not provide a secret, you cannot use secret-based verification for incoming webhook requests.

  <Note>
    The secret is write-only and is not returned by the API after creation. To change it later, use the [update workflow notification channel endpoint](/api-reference/api/workflow/update-workflow-notification-channel) instead.
  </Note>
</ParamField>

<ParamField type="object">
  Email delivery configuration. Required when `channel_type` is `email`.

  | Value             | Required | Description                                                                  |
  | ----------------- | -------- | ---------------------------------------------------------------------------- |
  | `recipient_email` | yes      | Email address to deliver notifications to.                                   |
  | `reply_to`        |          | Email address set as the reply-to header in notification emails.             |
  | `cc`              |          | Additional email addresses to copy on notification emails. Array of strings. |
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the channel.
</ResponseField>

<ResponseField name="channel_type" type="string">
  Channel type: `webhook` or `email`.
</ResponseField>

<ResponseField name="event_types" type="array">
  Subscribed event types.
</ResponseField>

<ResponseField name="enabled" type="boolean">
  Whether the channel is active.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the channel was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the channel was last updated.
</ResponseField>

<ResponseField name="description" type="string">
  Channel description.
</ResponseField>

<ResponseField name="url" type="string">
  Webhook endpoint URL. Present when `channel_type` is `webhook`.
</ResponseField>

<ResponseField name="email_config" type="object">
  Email delivery configuration. Present when `channel_type` is `email`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/notifications/channels" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "channel_type": "webhook",
      "url": "https://hooks.example.com/notify",
      "event_types": ["job.completed", "job.failed"],
      "description": "Job status alerts",
      "enabled": true
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsPostRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.create_workflow_channel(
      request=CreateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsPostRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          channel_type="webhook",
          url="https://hooks.example.com/notify",
          event_types=["job.completed", "job.failed"],
          description="Job status alerts",
          enabled=True,
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsPostRequest

  async def create_workflow_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.create_workflow_channel_async(
          request=CreateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsPostRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              channel_type="webhook",
              url="https://hooks.example.com/notify",
              event_types=["job.completed", "job.failed"],
              description="Job status alerts",
              enabled=True,
          )
      )
      print(response)

  asyncio.run(create_workflow_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
    "channel_type": "webhook",
    "description": "Job status alerts",
    "event_types": ["job.completed", "job.failed"],
    "enabled": true,
    "url": "https://hooks.example.com/notify",
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T10:00:00Z"
  }
  ```
</ResponseExample>


# Delete workflow
Source: https://docs.unstructured.io/api-reference/api/workflow/delete-workflow

DELETE /api/v1/workflows/{workflow_id}
Delete a workflow by its ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow to delete.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request DELETE \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteWorkflowRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.delete_workflow(
      request=DeleteWorkflowRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteWorkflowRequest

  async def delete_workflow():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.delete_workflow_async(
          request=DeleteWorkflowRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
          )
      )
      print(response)

  asyncio.run(delete_workflow())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Delete workflow notification channel
Source: https://docs.unstructured.io/api-reference/api/workflow/delete-workflow-notification-channel

DELETE /api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}
Delete workflow-scoped notification channel.

<Warning>
  Deleting a workflow-scoped notification channel is a permanent action and is not recoverable.
</Warning>

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

<ParamField type="string">
  The unique identifier of the notification channel to delete.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request DELETE \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdDeleteRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.delete_workflow_channel(
      request=DeleteWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdDeleteRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import DeleteWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdDeleteRequest

  async def delete_workflow_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.delete_workflow_channel_async(
          request=DeleteWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdDeleteRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          )
      )
      print(response)

  asyncio.run(delete_workflow_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Get workflow
Source: https://docs.unstructured.io/api-reference/api/workflow/get-workflow

GET /api/v1/workflows/{workflow_id}
Retrieve detailed information for a specific workflow by its ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the workflow.
</ResponseField>

<ResponseField name="name" type="string">
  Workflow name.
</ResponseField>

<ResponseField name="workflow_type" type="string">
  Workflow type: `custom` or `auto`.
</ResponseField>

<ResponseField name="status" type="string">
  Workflow state: `active`, `inactive`, or `paused`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the workflow was created.
</ResponseField>

<ResponseField name="source_id" type="string">
  Source connector ID.
</ResponseField>

<ResponseField name="destination_id" type="string">
  Destination connector ID.
</ResponseField>

<ResponseField name="schedule" type="string">
  Repeating run schedule.
</ResponseField>

<ResponseField name="workflow_nodes" type="array">
  Workflow processing pipeline nodes.

  For more information on workflow nodes, see [Workflow nodes](/api-reference/workflow/nodes/overview).
</ResponseField>

<ResponseField name="template_id" type="string">
  ID of the workflow template used to create this workflow.
</ResponseField>

<ResponseField name="reprocess_all" type="boolean">
  Whether all documents are reprocessed on every run.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the workflow was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetWorkflowRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.get_workflow(
      request=GetWorkflowRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetWorkflowRequest

  async def get_workflow():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.get_workflow_async(
          request=GetWorkflowRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
          )
      )
      print(response)

  asyncio.run(get_workflow())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
    "name": "my-workflow",
    "workflow_type": "auto",
    "status": "active",
    "source_id": "7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
    "destination_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "schedule": "daily",
    "workflow_nodes": null,
    "template_id": null,
    "reprocess_all": false,
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": null
  }
  ```
</ResponseExample>


# Get workflow notification channel
Source: https://docs.unstructured.io/api-reference/api/workflow/get-workflow-notification-channel

GET /api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}
Get workflow-scoped notification channel by ID.

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

<ParamField type="string">
  The unique identifier of the notification channel.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the channel.
</ResponseField>

<ResponseField name="channel_type" type="string">
  Channel type: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ResponseField>

<ResponseField name="event_types" type="array">
  Subscribed event types.
</ResponseField>

<ResponseField name="enabled" type="boolean">
  Whether the channel is active.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the channel was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the channel was last updated.
</ResponseField>

<ResponseField name="description" type="string">
  Channel description.
</ResponseField>

<ResponseField name="url" type="string">
  Webhook endpoint URL. Present when `channel_type` is `webhook`.
</ResponseField>

<ResponseField name="email_config" type="object">
  Email delivery configuration. Present when `channel_type` is `email`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.get_workflow_channel(
      request=GetWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdGetRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import GetWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdGetRequest

  async def get_workflow_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.get_workflow_channel_async(
          request=GetWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdGetRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          )
      )
      print(response)

  asyncio.run(get_workflow_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
    "channel_type": "webhook",
    "description": "Job status alerts",
    "event_types": ["job.completed", "job.failed"],
    "enabled": true,
    "url": "https://hooks.example.com/notify",
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T10:00:00Z"
  }
  ```
</ResponseExample>


# List workflow notification channels
Source: https://docs.unstructured.io/api-reference/api/workflow/list-workflow-notification-channels

GET /api/v1/workflows/{workflow_id}/notifications/channels
List workflow-scoped notification channels.

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

## Query parameters

<ParamField type="string">
  Channel type to filter by: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ParamField>

<ParamField type="boolean">
  Filter by enabled status.
</ParamField>

## Response

<ResponseField name="items" type="array">
  List of notification channel objects.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/notifications/channels?channel_type=webhook" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListWorkflowChannelsApiV1WorkflowsWorkflowIdNotificationsChannelsGetRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.list_workflow_channels(
      request=ListWorkflowChannelsApiV1WorkflowsWorkflowIdNotificationsChannelsGetRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          channel_type="webhook",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListWorkflowChannelsApiV1WorkflowsWorkflowIdNotificationsChannelsGetRequest

  async def list_workflow_channels():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.list_workflow_channels_async(
          request=ListWorkflowChannelsApiV1WorkflowsWorkflowIdNotificationsChannelsGetRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              channel_type="webhook",
          )
      )
      print(response)

  asyncio.run(list_workflow_channels())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "items": [
      {
        "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
        "channel_type": "webhook",
        "description": "Job status alerts",
        "event_types": ["job.completed", "job.failed"],
        "enabled": true,
        "url": "https://hooks.example.com/notify",
        "created_at": "2026-04-29T10:00:00Z",
        "updated_at": "2026-04-29T10:00:00Z"
      }
    ]
  }
  ```
</ResponseExample>


# List workflows
Source: https://docs.unstructured.io/api-reference/api/workflow/list-workflows

GET /api/v1/workflows/
Retrieve a list of workflows, optionally filtered by source, destination, state, name, date range, and supports pagination and sorting.

You can specify multiple query parameters, for example `?source_id=<connector-id>&status=<status>`.

## Query parameters

<ParamField type="string">
  Filter by source connector ID.
</ParamField>

<ParamField type="string">
  Filter by destination connector ID.
</ParamField>

<ParamField type="string">
  Filter by workflow state.

  | Value      | Description                                       |
  | ---------- | ------------------------------------------------- |
  | `active`   | Workflow is enabled and will run on its schedule. |
  | `inactive` | Workflow is disabled and will not run.            |
  | `paused`   | Workflow is temporarily paused.                   |
</ParamField>

<ParamField type="string">
  Filter by workflow name.
</ParamField>

<ParamField type="integer">
  Page number for pagination. Default: `1`.
</ParamField>

<ParamField type="integer">
  Results per page. Default: `20`.
</ParamField>

<ParamField type="string">
  Return workflows created after this ISO 8601 timestamp.
</ParamField>

<ParamField type="string">
  Return workflows created before this ISO 8601 timestamp.
</ParamField>

<ParamField type="string">
  Field to sort results by. Default: `id`.
</ParamField>

<ParamField type="string">
  Sort order. Default: `asc`.

  | Value  | Description       |
  | ------ | ----------------- |
  | `asc`  | Ascending order.  |
  | `desc` | Descending order. |
</ParamField>

<ParamField type="string">
  Filter by DAG node configuration ID.
</ParamField>

<ParamField type="boolean">
  If `true`, return only soft-deleted workflows. Default: `false`.
</ParamField>

<ParamField type="boolean">
  If `true`, include recommender system workflows. Default: `false`.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the workflow.
</ResponseField>

<ResponseField name="name" type="string">
  Workflow name.
</ResponseField>

<ResponseField name="workflow_type" type="string">
  Workflow type: `custom` or `auto`.
</ResponseField>

<ResponseField name="status" type="string">
  Workflow state: `active`, `inactive`, or `paused`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the workflow was created.
</ResponseField>

<ResponseField name="source_id" type="string">
  Source connector ID.
</ResponseField>

<ResponseField name="destination_id" type="string">
  Destination connector ID.
</ResponseField>

<ResponseField name="schedule" type="string">
  Repeating run schedule.
</ResponseField>

<ResponseField name="dag_nodes" type="array">
  Workflow processing pipeline nodes.

  For more information on workflow nodes, see [Workflow nodes](/api-reference/workflow/nodes/overview).
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the workflow was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request GET \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/?status=active&page=1&page_size=20" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListWorkflowsRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.list_workflows(
      request=ListWorkflowsRequest(
          status="active",
          page=1,
          page_size=20,
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import ListWorkflowsRequest

  async def list_workflows():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.list_workflows_async(
          request=ListWorkflowsRequest(
              status="active",
              page=1,
              page_size=20,
          )
      )
      print(response)

  asyncio.run(list_workflows())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  [
    {
      "id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
      "name": "my-workflow",
      "workflow_type": "auto",
      "status": "active",
      "source_id": "7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
      "destination_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      "schedule": "daily",
      "dag_nodes": null,
      "created_at": "2026-04-29T10:00:00Z",
      "updated_at": null
    }
  ]
  ```
</ResponseExample>


# Run workflow
Source: https://docs.unstructured.io/api-reference/api/workflow/run-workflow

POST /api/v1/workflows/{workflow_id}/run
Run a workflow manually, by triggering a new job if none is currently active.

<Note>
  This endpoint creates a workflow that persists until it is explicitly deleted (a *long-lived workflow*). To create a workflow that exists only for the duration of a single job run using local files as input, use the [create job endpoint](/api-reference/api/job/create-job) instead.
</Note>

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow to run.
</ParamField>

## Body

<ParamField type="array">
  One or more files to process at runtime. Provide as a multipart file upload.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the job.
</ResponseField>

<ResponseField name="workflow_id" type="string">
  ID of the workflow that triggered this job.
</ResponseField>

<ResponseField name="workflow_name" type="string">
  Name of the workflow that triggered this job.
</ResponseField>

<ResponseField name="status" type="string">
  Job status: `SCHEDULED`, `IN_PROGRESS`, `COMPLETED`, `STOPPED`, or `FAILED`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the job was created.
</ResponseField>

<ResponseField name="runtime" type="string">
  Job duration.
</ResponseField>

<ResponseField name="input_file_ids" type="array">
  IDs of input files submitted with the job.
</ResponseField>

<ResponseField name="output_node_files" type="array">
  Metadata about output files produced by the job.
</ResponseField>

<ResponseField name="job_type" type="string">
  Job type. Defaults to `ephemeral`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/run" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}"
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import RunWorkflowRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.run_workflow(
      request=RunWorkflowRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import RunWorkflowRequest

  async def run_workflow():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.run_workflow_async(
          request=RunWorkflowRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c"
          )
      )
      print(response)

  asyncio.run(run_workflow())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "workflow_id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
    "workflow_name": "my-workflow",
    "status": "SCHEDULED",
    "created_at": "2026-04-29T10:00:00Z",
    "runtime": null,
    "input_file_ids": null,
    "output_node_files": null,
    "job_type": "ephemeral"
  }
  ```
</ResponseExample>


# Update workflow
Source: https://docs.unstructured.io/api-reference/api/workflow/update-workflow

PUT /api/v1/workflows/{workflow_id}
Update an existing workflow's name, connectors, schedule, or workflow type.

<Note>
  Only long-lived workflows (workflows that persist until explicitly deleted) can be updated. Workflows created for the duration of a single [on-demand job run](/api-reference/api/job/create-job) cannot be updated.
</Note>

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow to update.
</ParamField>

## Body

<ParamField type="string">
  Updated workflow name.
</ParamField>

<ParamField type="string">
  Updated source connector ID.
</ParamField>

<ParamField type="string">
  Updated destination connector ID.
</ParamField>

<ParamField type="string">
  Updated execution mode: `auto` or `custom`.
</ParamField>

<ParamField type="string">
  Repeating run schedule. Valid values and their cron equivalents:

  | Value              | cron           | Description                                          |
  | ------------------ | -------------- | ---------------------------------------------------- |
  | `every 15 minutes` | `*/15 * * * *` | Every 15 minutes.                                    |
  | `every hour`       | `0 * * * *`    | At the first minute of every hour.                   |
  | `every 2 hours`    | `0 */2 * * *`  | At the first minute of every second hour.            |
  | `every 4 hours`    | `0 */4 * * *`  | At the first minute of every fourth hour.            |
  | `every 6 hours`    | `0 */6 * * *`  | At the first minute of every sixth hour.             |
  | `every 8 hours`    | `0 */8 * * *`  | At the first minute of every eighth hour.            |
  | `every 10 hours`   | `0 */10 * * *` | At the first minute of every tenth hour.             |
  | `every 12 hours`   | `0 */12 * * *` | At the first minute of every twelfth hour.           |
  | `daily`            | `0 0 * * *`    | At the first minute of every day.                    |
  | `weekly`           | `0 0 * * 0`    | At the first minute of every Sunday.                 |
  | `monthly`          | `0 0 1 * *`    | At the first minute of the first day of every month. |

  If omitted, the workflow does not automatically run on a repeating schedule.

  Workflows with a local source cannot be set to run on a repeating schedule.
</ParamField>

<ParamField type="array">
  Updated processing pipeline stages. Each node requires `id` (string, UUID) and `node_type` (string), and supports optional `node_subtype` (string), `config` (object), and `params` (object).

  For more information on workflow nodes, see [Workflow nodes](/api-reference/workflow/nodes/overview).
</ParamField>

<ParamField type="boolean">
  Default: `false`. If `true`, reprocesses all documents in the source location on every run. If `false`, the workflow excludes from future processing any files Unstructured determines are unchanged since the last time the workflow ran.

  Unstructured determines if a document has changed based on the document version. For each workflow, Unstructured maintains a record of documents (and their versions, if present) processed by that workflow. Each document record consists of:

  * A `record_id` derived from the document name and path.
  * A `record_version` derived from either the document Etag (if the source provider generates one) or the source provider's native version identifier.

  When you set `reprocess_all` to `false` for a source connector that supports `reprocess_all`, Unstructured uses this list of records to determine whether or not to process each document:

  * If the `record_id` does not exist in the workflow records, Unstructured processes the document.
  * If the `record_id` exists, but the `record_version` has changed, or there is no `record_version`, Unstructured processes the document.

  The following table lists out the possible `record-id` and `record_version` combinations, and the action Unstructured takes in each case:

  | `record_id` | `record_version` | Action              |
  | ----------- | ---------------- | ------------------- |
  | Exists      | Unchanged        | Do not process file |
  | Exists      | Changed          | Process file        |
  | Exists      | (none)           | Process file        |
  | New         | (Does not apply) | Process file        |

  <Note>
    Renaming a document results in a new `record_id`; Unstructured will then reprocess the renamed document when the workflow runs.
  </Note>

  The following table lists the source connectors that support the `reprocess_all` setting. The **Record version base** column specifies the versioning information Unstructured uses to generated the corresponding record version for each processed document.

  Source connectors that do not support `reprocess_all` reprocess every document in the source location each time the workflow runs.

  | Connector                                                                | `record_version` base |
  | ------------------------------------------------------------------------ | --------------------- |
  | [Amazon S3](/api-reference/workflow/sources/s3)                          | ETag                  |
  | [Azure Blob Storage](/api-reference/workflow/sources/azure-blob-storage) | ETag                  |
  | [Box](/api-reference/workflow/sources/box)                               | Provider version ID   |
  | [Dropbox](/api-reference/workflow/sources/dropbox)                       | Provider version ID   |
  | [Elastisearch](/api-reference/workflow/sources/elasticsearch)            | Provider version ID   |
  | [Google Cloud Storage](/api-reference/workflow/sources/google-cloud)     | ETag                  |
  | [Google Drive](/api-reference/workflow/sources/google-drive)             | Provider version ID   |
  | [Microsoft OneDrive](/api-reference/workflow/sources/onedrive)           | Provider version ID   |
  | [Microsoft SharePoint](/api-reference/workflow/sources/sharepoint)       | Provider version ID   |

  Additional considerations to take into account when setting `reprocess_all` to `false`:

  * Unstructured only adds document records for documents that it successfully processes. Documents that failed to process will be reprocessed the next time the workflow is run.
  * Because S3 ETags are content-based, changing the metadata on an S3 object will not result in it being reprocessed.
  * For source providers that support the S3 protocol, be aware that deleting an object and then reuploading it to the source location will maintain the same `record_id`, but may result in a different `record_version` being generated. This is especially true of multipart uploads. This results in Unstructured reprocessing the document.
  * For source providers that offer Key Management Services (KMS), be aware that server-side encryption can change document ETags. This results in the the `record_version` of a document changing, and Unstructured reprocessing the document.
  * If you clone or recreate a source connector, the resulting connector does not include the document processing history of the previous connector.
  * Changing a workflow's configuration does not automatically result in Unstructured reprocessing all documents. For example, changing chunker, embedder, enrichment, or partitioner settings may not result in reprocessing all document. To reprocess all documents using new workflow settings, set `reprocess_all` to `true` for at least the next workflow run.
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the workflow.
</ResponseField>

<ResponseField name="name" type="string">
  Workflow name.
</ResponseField>

<ResponseField name="workflow_type" type="string">
  Workflow type: `custom` or `auto`.
</ResponseField>

<ResponseField name="status" type="string">
  Workflow state: `active`, `inactive`, or `paused`.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the workflow was created.
</ResponseField>

<ResponseField name="source_id" type="string">
  Source connector ID.
</ResponseField>

<ResponseField name="destination_id" type="string">
  Destination connector ID.
</ResponseField>

<ResponseField name="schedule" type="string">
  Repeating run schedule.
</ResponseField>

<ResponseField name="dag_nodes" type="array">
  Workflow processing pipeline nodes.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the workflow was last updated.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request PUT \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "name": "my-updated-workflow",
      "schedule": "weekly"
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateWorkflowRequest
  from unstructured_client.models.shared import UpdateWorkflow

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.update_workflow(
      request=UpdateWorkflowRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          update_workflow=UpdateWorkflow(
              name="my-updated-workflow",
              schedule="weekly",
          ),
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateWorkflowRequest
  from unstructured_client.models.shared import UpdateWorkflow

  async def update_workflow():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.update_workflow_async(
          request=UpdateWorkflowRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              update_workflow=UpdateWorkflow(
                  name="my-updated-workflow",
                  schedule="weekly",
              ),
          )
      )
      print(response)

  asyncio.run(update_workflow())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
    "name": "my-updated-workflow",
    "workflow_type": "auto",
    "status": "active",
    "source_id": "7f3e2a1b-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
    "destination_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "schedule": "weekly",
    "dag_nodes": null,
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T11:00:00Z"
  }
  ```
</ResponseExample>


# Update workflow notification channel
Source: https://docs.unstructured.io/api-reference/api/workflow/update-workflow-notification-channel

PATCH /api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}
Update workflow-scoped notification channel. Only provided fields are updated.

Channels that have not been [verified](/api-reference/api/workflow/verify-workflow-notification-channel) cannot be updated.

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

<ParamField type="string">
  The unique identifier of the notification channel to update.
</ParamField>

## Body

<ParamField type="string">
  Updated channel description. Maximum 255 characters.
</ParamField>

<ParamField type="array">
  Updated list of event types to subscribe to.

  | Value             | Description                                           |
  | ----------------- | ----------------------------------------------------- |
  | `job.scheduled`   | Job queued to run                                     |
  | `job.in_progress` | Job has started                                       |
  | `job.completed`   | Job finished processing                               |
  | `job.stopped`     | Job has stopped                                       |
  | `job.failed`      | Job failed to initialize without processing any files |
</ParamField>

<ParamField type="boolean">
  Whether the channel is active.
</ParamField>

<ParamField type="string">
  Updated webhook endpoint URL. Applies when `channel_type` is `webhook`.
</ParamField>

<ParamField type="string">
  Updated signing secret for verifying incoming webhook requests. Must be between 24 and 75 bytes (24 to 75 ASCII characters). Applies when `channel_type` is `webhook`. Omit to keep the current value.

  Use this secret to [verify incoming webhook requests](/api-reference/webhooks#verify-webhook-requests). If you do not provide a secret, you cannot use secret-based verification for incoming webhook requests.
</ParamField>

<ParamField type="object">
  Updated email delivery configuration. Applies when `channel_type` is `email`.

  | Value             | Required | Description                                                                  |
  | ----------------- | -------- | ---------------------------------------------------------------------------- |
  | `recipient_email` | yes      | Email address to deliver notifications to.                                   |
  | `reply_to`        |          | Email address set as the reply-to header in notification emails.             |
  | `cc`              |          | Additional email addresses to copy on notification emails. Array of strings. |
</ParamField>

## Response

<ResponseField name="id" type="string">
  Unique identifier for the channel.
</ResponseField>

<ResponseField name="channel_type" type="string">
  Channel type: `webhook` or `email`.

  For more information, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).
</ResponseField>

<ResponseField name="event_types" type="array">
  Subscribed event types.
</ResponseField>

<ResponseField name="enabled" type="boolean">
  Whether the channel is active.
</ResponseField>

<ResponseField name="created_at" type="string">
  ISO 8601 timestamp when the channel was created.
</ResponseField>

<ResponseField name="updated_at" type="string">
  ISO 8601 timestamp when the channel was last updated.
</ResponseField>

<ResponseField name="description" type="string">
  Channel description.
</ResponseField>

<ResponseField name="url" type="string">
  Webhook endpoint URL. Present when `channel_type` is `webhook`.
</ResponseField>

<ResponseField name="email_config" type="object">
  Email delivery configuration. Present when `channel_type` is `email`.
</ResponseField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request PATCH \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "event_types": ["job.completed", "job.failed", "job.in_progress"],
      "enabled": true
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdPatchRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.update_workflow_channel(
      request=UpdateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdPatchRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          event_types=["job.completed", "job.failed", "job.in_progress"],
          enabled=True,
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import UpdateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdPatchRequest

  async def update_workflow_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.update_workflow_channel_async(
          request=UpdateWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdPatchRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
              event_types=["job.completed", "job.failed", "job.in_progress"],
              enabled=True,
          )
      )
      print(response)

  asyncio.run(update_workflow_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {
    "id": "c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
    "channel_type": "webhook",
    "description": "Job status alerts",
    "event_types": ["job.completed", "job.failed", "job.in_progress"],
    "enabled": true,
    "url": "https://hooks.example.com/notify",
    "created_at": "2026-04-29T10:00:00Z",
    "updated_at": "2026-04-29T11:00:00Z"
  }
  ```
</ResponseExample>


# Verify workflow notification channel
Source: https://docs.unstructured.io/api-reference/api/workflow/verify-workflow-notification-channel

POST /api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}/verify
Verify a workflow-scoped notification channel using a verification code.

<Note>
  Email channels require verification. A 6-digit verification code is sent to the recipient email address when the channel is created. Webhook channels do not require verification.

  For more information, see [Verify the email channel](/api-reference/email#verify-the-email-channel).
</Note>

## Path parameters

<ParamField type="string">
  The unique identifier of the workflow.
</ParamField>

<ParamField type="string">
  The unique identifier of the notification channel to verify.
</ParamField>

## Body

<ParamField type="string">
  The 6-digit verification code sent to the recipient email address during channel creation.
</ParamField>

<RequestExample>
  ```bash cURL theme={null}
  curl --request POST \
    --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c/notifications/channels/c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f/verify" \
    --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
    --header "Content-Type: application/json" \
    --data '{
      "verification_code": "123456"
    }'
  ```

  ```python Python SDK theme={null}
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import VerifyWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdVerifyPostRequest

  client = UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
      server_url=os.getenv("UNSTRUCTURED_API_URL"),
  )

  response = client.workflows.verify_workflow_channel(
      request=VerifyWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdVerifyPostRequest(
          workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
          channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
          verification_code="123456",
      )
  )
  print(response)
  ```

  ```python Python SDK (async) theme={null}
  import asyncio
  import os
  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import VerifyWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdVerifyPostRequest

  async def verify_workflow_channel():
      client = UnstructuredClient(
          api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
          server_url=os.getenv("UNSTRUCTURED_API_URL"),
      )
      response = await client.workflows.verify_workflow_channel_async(
          request=VerifyWorkflowChannelApiV1WorkflowsWorkflowIdNotificationsChannelsChannelIdVerifyPostRequest(
              workflow_id="f0a1b2c3-4d5e-6f7a-8b9c-0d1e2f3a4b5c",
              channel_id="c1d2e3f4-5a6b-7c8d-9e0f-1a2b3c4d5e6f",
              verification_code="123456",
          )
      )
      print(response)

  asyncio.run(verify_workflow_channel())
  ```
</RequestExample>

<ResponseExample>
  ```json Response theme={null}
  {}
  ```
</ResponseExample>


# Workflow endpoints
Source: https://docs.unstructured.io/api-reference/api/workflow/workflow-apis



Build and operate the end-to-end processing workflows that move documents from source to destination. Create, configure, and trigger runs, and manage workflow-scoped notification channels to stay informed on execution status.

| Name                                                                                                     | Method   | Path                                                                         |
| -------------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------- |
| [List workflows](/api-reference/api/workflow/list-workflows)                                             | `GET`    | `/api/v1/workflows/`                                                         |
| [Get workflow](/api-reference/api/workflow/get-workflow)                                                 | `GET`    | `/api/v1/workflows/{workflow_id}`                                            |
| [Create workflow](/api-reference/api/workflow/create-workflow)                                           | `POST`   | `/api/v1/workflows/`                                                         |
| [Update workflow](/api-reference/api/workflow/update-workflow)                                           | `PUT`    | `/api/v1/workflows/{workflow_id}`                                            |
| [Delete workflow](/api-reference/api/workflow/delete-workflow)                                           | `DELETE` | `/api/v1/workflows/{workflow_id}`                                            |
| [Run workflow](/api-reference/api/workflow/run-workflow)                                                 | `POST`   | `/api/v1/workflows/{workflow_id}/run`                                        |
| [List workflow notification channels](/api-reference/api/workflow/list-workflow-notification-channels)   | `GET`    | `/api/v1/workflows/{workflow_id}/notifications/channels`                     |
| [Get workflow notification channel](/api-reference/api/workflow/get-workflow-notification-channel)       | `GET`    | `/api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}`        |
| [Create workflow notification channel](/api-reference/api/workflow/create-workflow-notification-channel) | `POST`   | `/api/v1/workflows/{workflow_id}/notifications/channels`                     |
| [Update workflow notification channel](/api-reference/api/workflow/update-workflow-notification-channel) | `PATCH`  | `/api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}`        |
| [Delete workflow notification channel](/api-reference/api/workflow/delete-workflow-notification-channel) | `DELETE` | `/api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}`        |
| [Verify workflow notification channel](/api-reference/api/workflow/verify-workflow-notification-channel) | `POST`   | `/api/v1/workflows/{workflow_id}/notifications/channels/{channel_id}/verify` |

To read and manage *workspace-scoped* notification channels, use the following endpoints:

* [List workspace notification channels](/api-reference/api/notification/list-notification-channels)
* [Get workspace notification channel](/api-reference/api/notification/get-notification-channel)
* [Create workspace notification channel](/api-reference/api/notification/create-notification-channel)
* [Update workspace notification channel](/api-reference/api/notification/update-notification-channel)
* [Delete workspace notification channel](/api-reference/api/notification/delete-notification-channel)
* [Verify workspace notification channel](/api-reference/api/notification/verify-notification-channel)

For more information about notification channels, see [Webhooks](/api-reference/webhooks) and [Email notifications](/api-reference/email).


# Email Notifications
Source: https://docs.unstructured.io/api-reference/email

Learn how to create and verify email notification channels to receive automated emails for Unstructured workflow job events.

Email notification channels are endpoints that send automated emails when workflow job events occur. Similar to [webhooks](/api-reference/webhooks), they allow you to automate notifications to specified recipients.

Email notification channels are only available through the Unstructured API.

You can create email notification channels at the workspace or workflow level:

* *Workspace-scoped* notification channels send emails when the selected events occur on jobs for all workflows in the workspace.
* *Workflow-scoped* notification channels send emails when the selected events occur on jobs for just the specified workflow.

Email notification channels support the following event types:

| Event type        | Description                                           |
| ----------------- | ----------------------------------------------------- |
| `job.scheduled`   | Job queued to run                                     |
| `job.in_progress` | Job has started                                       |
| `job.completed`   | Job finished processing                               |
| `job.stopped`     | Job has stopped                                       |
| `job.failed`      | Job failed to initialize without processing any files |

The following 3-minute video provides an overview of email notification channels:

<iframe title="YouTube video player" />

The following 5-minute video shows how to use the Unstructured API to create and verify email notification channels:

<iframe title="YouTube video player" />

## Create the email channel

To create an email notification channel, use the Unstructured API:

* For workspace-scoped channels, use [Create notification channel](/api-reference/api/notification/create-notification-channel).
* For workflow-scoped channels, use [Create workflow notification channel](/api-reference/api/workflow/create-workflow-notification-channel).

In each case, the response will include the unique identifier for the notification channel, which you will use to [verify the channel](/api-reference/email#verify-the-email-channel).

```json theme={null}
{
    "channel_type": "email",
    "id": "ep_0a1B2c3D4e5F6g7H8i9J0k1L2m3N",
    . . . 
    "enabled": false,
    . . . 
}
```

Once the notification channel is created, Unstructured will send a verification code to the recipient email address. You will need this code to verify the channel.

The following example creates an email notification channel that sends notifications for all job status events to the specified email address:

```bash cURL theme={null}
curl --request POST \
 --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels" \
 --header "Content-Type: application/json" \
 --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
 --data '{
    "channel_type": "email",
    "description": "Workspace level job events",
    "event_types": ["job.completed", "job.scheduled", "job.in_progress", "job.stopped", "job.failed"],
    "email_config": {
      "recipient_email": "donotreply@example.com"
    }
  }'
```

The following example creates an email notification channel that sends notifications only for the specified workflow, and just for jobs that have failed:

```bash cURL theme={null}
curl --request POST \
  --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/12345678-1234-1234-1234-123456789012/notifications/channels" \
  --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
  --header "Content-Type: application/json" \
  --data '{
    "channel_type": "email",
    "event_types": ["job.failed"],
    "description": "Workflow-specific status alerts",
    "email_config": {
      "recipient_email": "donotreply@example.com"
    }
  }'
```

## Verify the email channel

Unstructured creates email notification channels in a disabled state, and they require verification before they can receive notifications. To verify a channel, use the Unstructured API:

* For workspace-scoped channels, use [Verify notification channel](/api-reference/api/notification/verify-notification-channel).
* For workflow-scoped channels, use [Verify workflow notification channel](/api-reference/api/workflow/verify-workflow-notification-channel).

To use these APIs, you need two pieces of information:

* The channel ID, which is included in the response when you create the channel.
* The verification code, which is a six-digit code sent to the recipient email address when you create the channel.

A response with an HTTP status code of `204 No Content` indicates that the channel is now verified and active.

The following example verifies a workspace-scoped email notification channel:

```bash cURL theme={null}
curl --request POST \
  --url "${UNSTRUCTURED_API_URL}/api/v1/notifications/channels/ep_9Z8y7X6w5V4u3T2s1R0q9P8o7N6m/verify" \
  --header "Content-Type: application/json" \
  --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
  --data '{
    "code": "123456"
  }'
```

The following example verifies a workflow-scoped email notification channel:

```bash cURL theme={null}
curl --request POST \
  --url "${UNSTRUCTURED_API_URL}/api/v1/workflows/12345678-1234-1234-1234-123456789012/notifications/channels/ep_1a2B3c4D5e6F7g8H9i0J1k2L3m4N/verify" \
  --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
  --header "Content-Type: application/json" \
  --data '{
    "code": "123456"
  }'
```

## Test the email channel

To test an email notification channel, run a workflow job and check your email inbox for the specified notifications.


# Unstructured API on AWS
Source: https://docs.unstructured.io/api-reference/legacy-api/aws

Follow these steps to deploy the Unstructured API service into your AWS account.

<Warning>
  The Unstructured API on AWS is deprecated. It is no longer supported and is not being actively updated.
  Unstructured is now available on the AWS Marketplace as a private offering. To explore supported options
  for running Unstructured within your virtual private cloud (VPC), email Unstructured Sales at
  [sales@unstructured.io](mailto:sales@unstructured.io).

  This page is not being actively updated. It might contain out-of-date information. This page is provided for legacy reference purposes only.
</Warning>

<Warning>
  This article describes how to create several interrelated resources in your AWS account.
  Your AWS account will be charged on an ongoing basis for these resources, even if you are not actively using them.<br /><br />
  Manually stopping or terminating the associated Amazon EC2 instances alone will not reduce these ongoing charges.<br /><br />
  To stop accruing all related ongoing charges, you must delete all of the associated AWS resources.
  To do this, see [Manage related AWS account costs](#manage-related-aws-account-costs).
</Warning>

*Estimated time to complete: 30 minutes*

The requirements are as follows.

1. **An AWS account**:

   * If you have an existing account, log in: [https://aws.amazon.com/](https://aws.amazon.com) > **Sign In to the Console**.

   * If you do not have an existing account, create one: [https://aws.amazon.com/free](https://aws.amazon.com/free) > **Create a Free Account**.

2. **IAM permissions**: In a later step, AWS CloudFormation creates required infrastructure in your account. To learn how to create the associated role, see [Creating IAM Roles with AWS CloudFormation](https://blog.awsfundamentals.com/aws-iam-roles-with-aws-cloudformation#heading-creating-iam-roles-with-aws-cloudformation).

3. **SSH key pair**: For secure access to the Amazon EC2 instance that CloudFormation creates in a later step, create an SSH key pair. To learn how, see [Create a key pair for your Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html).

## Part I: Setting up the Virtual Private Cloud (VPC)

*Note: If you have already configured a Virtual Private Cloud (VPC) for your organization that meets the requirements for deploying the Unstructured API, you may skip this part and proceed to the Part II. Ensure that your existing VPC setup includes the necessary subnets, internet gateway, and route tables as outlined in this guide.*

In Part I, you will construct a resilient and secure infrastructure within AWS by setting up a Virtual Private Cloud (VPC). Your VPC will encompass a dual-tiered subnet model consisting of both **public** and **private** subnets across multiple Availability Zones (AZs).

You will establish the foundational network structure for deploying the Unstructured API by creating two public subnets and one private subnet within your VPC. The public subnets will host resources that require direct access to the internet, such as a load balancer, enabling them to communicate with external users. The private subnet is designed for resources that should not be directly accessible from the internet, like EC2 Compute Engine.

<img alt="Infrastructure Diagram" />

1. **Access the VPC dashboard**:

   a.   In the AWS Management Console, in the top menu bar, click **Services > Networking & Content Delivery > VPC**.<br />

   b.   In the sidebar, click **Your VPCs**, and then click **Create VPC**.<br />

2. **Create the VPC**:

   a.   Select **VPC only**.<br />

   b.   Enter a **Name tag** for your VPC.<br />

   c.   Specify the **IPv4 CIDR block** (for example, `10.0.0.0/16`).<br />

   d.   You may leave **IPv6 CIDR block**, **Tenancy**, and **Tags** settings at their defaults.<br />

   e.   Click **Create VPC**.

<img alt="create vpc" />

3. **Create the subnets**:

   a.   After creating the VPC, in the sidebar, click **Subnets**.<br />

   b.   Click **Create subnet**.<br />

   c.   In the **VPC ID** dropdown menu. select the VPC that you just created.<br />

   d.   For the first public subnet:<br />

   * Enter a **Subnet name**.

   * Select an **Availability Zone**.

   * Specify the **IPv4 CIDR block** (for example, `10.0.0.0/16`).

   * Specify the **IPv4 subnet CIDR block** (for example, `10.0.1.0/24`).

   * You may leave the **Tags** setting at its default.

   * Click **Add new subnet**. (Do not click **Create subnet** yet.)

   e.  Repeat the process for the second public subnet with a different **Availability Zone** and **IPv4 subnet CIDR block** (for example, `10.0.2.0/24`).<br />

   * *Note: Each subnet must reside entirely within one Availability Zone and cannot span zones. If you specify the same Availability Zone or IPv4 subnet CIDR block as the first public subnet, AWS CloudFormation might fail in a later step*.

   * To learn more, see [Subnet basics](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html#subnet-basics).

   * Click **Add new subnet**. (Do not click **Create subnet** yet.)

   f.  Repeat the process for the private subnet with a different **Availability Zone** and **IPv4 subnet CIDR block** (for example, `10.0.3.0/24`).<br />

   * *Note: Each subnet must reside entirely within one Availability Zone and cannot span zones. If you specify the same Availability Zone or IPv4 subnet CIDR block as the first or second public subnets, AWS CloudFormation might fail in a later step*.

   g.  Click **Create subnet**.

<img alt="create subnet" />

4. **Create the internet gateway (for the public subnets)**:

   a.   In the sidebar, click **Internet gateways**.<br />

   b.   Click **Create internet gateway**, enter a **Name tag**, and click **Create internet gateway**.<br />

   c.   In the sidebar, click **Internet gateways** again.<br />

   d.   Click the **Internet gateway ID** for the internet gateway that you just created.<br />

   e.   Click **Actions > Attach to VPC**.<br />

   f.   In the **Available VPCs** dropdown list, select the VPC from *Step 2 - Create the VPC*.<br />

   g.   Click **Attach internet gateway**.<br />

<img alt="create internet gateway" />

5. **Set up route tables (for the public subnets)**:

   AWS automatically created a default route table in *Step 3 - Create the subnets*. To tailor your network architecture, you will create a new route table specifically for your public subnets, which will include a route to the internet gateway from *Step 4 - Create the internet gateway (for the public subnets)*.

   a.   In the sidebar, click *Route tables*.

   b.   Click **Create route table**.

   c.   Enter a **Name**.

   d.   Select the **VPC** from *Step 2 - Create the VPC*.

   e.   Click **Create route table**.

<img alt="create route table" />

6. **Associate public subnets to the route table and internet gateway**:

   a.  Connect the **public subnets** to the **route table** from *Step 5 - Set up route tables (for the public subnets)*:<br />

   * In the sidebar, click **Subnets**.

   * Select the first public subnet from *Step 3 - Create the subnets*.

   * Click **Actions > Edit route table association**.

   * In the **Route table ID** dropdown list, select the route table from *Step 5 - Set up route tables (for the public subnets)*, and then click **Save**.

   * Repeat the process for the second public subnet.

   b.   Now, you’ll ensure that the two public subnets can access the internet by connecting the route table to the internet gateway:<br />

   * In the sidebar, click **Route tables**.

   * Select the route table from *Step 5 - Set up route tables (for the public subnets)*.

   * Click **Actions > Edit routes**.

   * Click **Add route**, in the destination box, enter `0.0.0.0/0`, which represents all IP addresses. In the **Target** dropdown list, select **Internet Gateway**, and select the internet gateway from *Step 4 - Create the internet gateway (for the public subnets)*.

   * Click **Save changes** to establish the route, granting internet access to the first and second public subnets at the same time.

   c.  For the **private subnet**:

   * In the sidebar, click **Subnets**.

   * Select the private subnet from *Step 3 - Create the subnets*.

   * Click **Actions > Edit route table association**.

   * In the **Route table ID** dropdown list, select the main route table, or create and then select a new route table without a route to the internet gateway.

   * Click **Save**.

<img alt="connect public subnet to route table" />

<img alt="edit routes" />

7. **Inspect the VPC resource map**:

   You can check the configurations from the resource maps on the VPC details dashboard by clicking **Your VPCs** in the sidebar, clicking the **VPC ID** for your VPC, and then clicking the **Resource map** tab.

<img alt="VPC Resource Maps" />

## Part II: Deploying the Unstructured API from the AWS Marketplace

8. **Go to the Unstructured API page on AWS Marketplace**:

   a.   Leaving the VPC dashboard from Part I open, in a separate web browser tab, go to the [Unstructured API](http://aws.amazon.com/marketplace/pp/prodview-fuvslrofyuato) product page in the AWS Marketplace.

   b.   Click **Continue to Subscribe**.

   c.   Review the terms and conditions.

   d.   Click **Continue to Configuration**.

<img alt="Unstructured API on AWS Marketplace" />

9. **Configure the CloudFormation template**:

   a.   In the **Fulfillment option** dropdown list, select **CloudFormation Template**.

   b.   For **Fulfillment option** and **Software version**, leave the default `UnstructuredAPI` template and software version.

   c.   In the **Region** dropdown list, select the Region that corresponds to the VPC from Part I.

   * *Note: You must select the same Region where you set up the VPC in Part I. To find the Region, on the VPC dashboard tab from Part I that you left open, with your VPC displayed, find the VPC's Region name next to your username in the top navigation bar.*

   d.   Click **Continue to Launch**.

   e.   In the **Choose Action** dropdown list, select **Launch CloudFormation**.

   f.   Click **Launch**.

<img alt="CloudFormation Configuration" />

10. **Create the CloudFormation stack**:

After you click **Launch**, the **Create stack** page appears in CloudFormation.

**Step 1: Create the stack**

a.   Leave **Choose an existing template** selected.

b.   Leave **Amazon S3 URL** selected and the default **Amazon S3 URL** value unchanged.

c.   Click **Next**.

<img alt="Create Stack" />

**Step 2: Specify the stack's details**

a.   Enter some unique **Stack name**.

b.   In the **Parameters** section, in the **InstanceType** drop-down list, select **m5.xlarge**.

c.   In the **KeyName** drop-down list, select the name of the SSH key pair from the beginning of this article.

d.   In the **LoadBalancerScheme** dropdown list, select **internet-facing**.

e.   For **SSHLocation**, enter `0.0.0.0/0`, but only if you allow public access on the internet.

* **Note**: It is generally recommended to limit SSH access to a specific IP range for enhanced security. This can be done by setting the `SSHLocation` to the IP address or range associated with your organization. Please consult your IT department or VPN vendor to obtain the correct IP information for these settings.

* AWS provides `AWS Client VPN`, which is a managed client-based VPN service that enables secure access AWS resources and resources in your on-premises network. To learn more, see [Getting started with AWS Client VPN](https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-getting-started.html).

f.   In the **Subnets** dropdown multiselect list, select the two public subnets and the private subnet from Part I.

g.   In the **VPC** dropdown list, select the VPC from Part I.

h.   You can leave the default values for all of the other **Parameters** fields.

i.   Click **Next**.

<img alt="Specify stack details" />

**Step 3: Configure the stack's options**

a.   You can leave the default values, or specify any non-default stack options.

b.   Click **Next**.

<img alt="Specify stack options" />

**Step 4: Review**

a.   Review the stack's settings.

b.   Click **Submit**.

<img alt="Review stack" />

11. **Get the Unstructured API endpoint**:

a.   The CloudFormation details page for the stack appears. If you do not see it, on the sidebar, click **Stacks**, and then click the name of your stack.

b.   Check the status of the CloudFormation stack. A successful deployment will show a **CREATE\_COMPLETE** value for the **Status** field on the **Stack Info** tab on this stack's details page. The deployment can take several minutes.

c.   After a successful deployment, click the **Resources** tab on this stack's details page. Then click the **Physical ID** link next to **ApplicationLoadBalancer** on this tab.

d.   On the **EC2 > Load balancers > (Load balancer ID)** page that appears, copy the **DNS Name** value, which is shown as an **(A Record)** and ends with `.elb.amazonaws.com`.

* Note: You will use this **DNS Name** to replace the `<application-load-balancer-dns-name>` for the following healthcheck and data processing steps.

<img alt="Unstructured API Endpoint" />

## Healthcheck

Perform a health check by running this [curl](https://curl.se/) command from a terminal on your local machine, replacing `<application-load-balancer-dns-name>` with your application load balancer's DNS name. This health check can take several minutes:

```bash theme={null}
curl http://<application-load-balancer-dns-name>/healthcheck

```

<img alt="Healthcheck" />

## Data processing

For example, run one of the following, setting the following environment variables to make your code more portable:

* Set `UNSTRUCTURED_API_URL` to `http://`, followed by your load balancer's DNS name, followed by `/general/v0/general`.

  <Info>You can now use this value (`http://`, followed by your load balancer's DNS name, followed by `/general/v0/general`) in place of
  calling the [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) URL as described elsewhere in the Unstructured API documentation.</Info>

* Set `LOCAL_FILE_INPUT_DIR` to the path on your local machine to the files for the Unstructured API to process. If you do not have any input files available, you can download any of the ones from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in GitHub.

* Set `LOCAL_FILE_OUTPUT_DIR` to the path on your local machine for Unstructured API to send the processed output in JSON format:

<AccordionGroup>
  <Accordion title="Ingest CLI">
    You must first [install the Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the command-line option `--api-key` Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the command-line option `--api-key`.

    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python library">
    You must first [install the Unstructured Ingest Python library](/open-source/ingestion/python-ingest).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the parameter `api_key`. Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the parameter `api_key`.

    ```python Python Ingest theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

## Accessing the hosting EC2 instance

If you need to access the Amazon EC2 instance that hosts the Unstructured API, do the following:

1. In the CloudFormation console, open the details page for the stack from Part II. If you do not see it, on the CloudFormation console's sidebar, click **Stacks**, and then click the name of your stack.

2. Click the **Resources** tab on this stack's details page. Then click the **Physical ID** link next to **EC2TargetGroup** on this tab.

3. On the **EC2 > Target groups > (CloudFormation stack name)** page that appears, on the **Targets** tab, click the **Instance ID** link.

4. In the list of instances that appears, click the **Instance ID** link.

5. Click **Connect**, and then follow any of the on-screen options to access the EC2 instance.

## Manage related AWS account costs

After you run the CloudFormation stack that you created in Part II, charges will begin accruing to your AWS account on an ongoing basis for related AWS resources.
The amounts of these charges vary based on where these resource are located, which resources are covered by AWS Free Tier offerings, the extent to
which you customize these resources' settings, how much you use these resources, and other factors. Stopping or terminating
the related Amazon EC2 instances alone will not eliminate these ongoing charges.

To stop these charges from accruing,
[delete the CloudFormation stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html)
that you created and ran in Part II. This stops and deletes all of the related AWS resources.

Before you delete the stack, note the following:

* You should click the **Resources** tab on the stack's details page to be aware of the associated resources that will be deleted.
* You should note any resource dependencies, resources with deletion protection or termination protection enabled, or nested stacks
  that might prevent stack deletion, and resolve these issues that could prevent stack deletion. To find these kinds of issues:

  * On the **Template** tab on the stack's details page, look for occurences of the `DependsOn` attribute, which are set to the name
    of the resource dependency.
  * On the **Template** tab on the stack's details page, look for occurences of the `DeletionPolicy` attribute set to `Retain` or the
    `UpdateReplacePolicy` attribute set to `Retain`. The associated resources have deletion protection enabled.
  * On the **Stack info** tab on the stack's details page, look for the **Termination protection** field. If it is set to **Activated**,
    termination protection is enabled.
  * On the **Resources** tab on the stack's details page, look for resources with their **Type** set to `AWS::CloudFormation::Stack`. These indicate nested stacks.

After you delete the stack, you should check your [AWS Billing and Cost Management dashboard](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/billinginfo.html) to confirm that associated charges are no longer accruing.


# Unstructured API on Azure
Source: https://docs.unstructured.io/api-reference/legacy-api/azure



<Warning>
  The Unstructured API on Azure is deprecated. It is no longer supported and is not being actively updated.
  Unstructured is now available on the Azure Marketplace as a private offering. To explore supported options
  for running Unstructured within your virtual private cloud (VPC), email Unstructured Sales at
  [sales@unstructured.io](mailto:sales@unstructured.io).

  This page is not being actively updated. It might contain out-of-date information. This page is provided for legacy reference purposes only.
</Warning>

Follow these steps to deploy the Unstructured API service into your Azure account.

<Warning>
  This article describes how to create several interrelated resources in your Azure account.
  Your Azure account will be charged on an ongoing basis for these resources, even if you are not actively using them.<br /><br />
  Manually shutting down the associated Azure virtual machine when you are not using it can help reduce—but not fully eliminate—these ongoing charges.<br /><br />
  To stop accruing all related ongoing charges, you must delete all of the associated Azure resources.
</Warning>

<Steps>
  <Step title=" Log in to the Azure Portal">
    Go to [https://portal.azure.com](https://portal.azure.com/).
  </Step>

  <Step title="Access the Azure Marketplace">
    Go to the [Unstructured Data Preprocessing - Customer Hosted API](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/unstructured1691024866136.customer_api_v1?tab=Overview/) offering in the Azure Marketplace.

    <img alt="Azure Marketplace" />
  </Step>

  <Step title="Start the deployment process">
    1. Click **Get It Now** and fill out the form.
    2. Read the terms and click **Continue**.
    3. Click **Create**.

       <img alt="Deployment Process" />
  </Step>

  <Step title=" Configure the deployment options">
    1. On the **Create a virtual machine** page, click the **Basics** tab.

    2. In the **Project details** section, select an existing **Subscription**, and select an **Resource group** from the dropdown menus, or create a new resource group by clicking **Create new**.

       <img alt="project details" />

    3. In the **Instance details** section, enter a name in the **Virtual machine name** field. Note this name, as you will need it later steps.

    4. Select a **Region** from the dropdown menu.

    5. For **Image**, select **Unstructured Customer Hosted API Hourly - x64 Gen2** (*default*).

    6. For **Size**, select a VM size from the dropdown menu, or leave the default VM size selection. To learn more, see [Azure VM comparisons](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/).

       <img alt="instance details" />

    7. In the **Administrator account** section, for **Authentication type**, select **SSH public key** or **Password**.

    8. Enter the credential settings, depending on the authentication type.

       <img alt="administrator account" />

    <Note>Do not click **Review + create** yet. You must still set up the load balancer.</Note>
  </Step>

  <Step title="Set up the load balancer">
    1. Click the **Networking** tab.

    2. In the **Networking interface** section, fill out the following fields:

       * **Virtual network**: Click **Create new**, or select an existing virtual network from the dropdown menu. To learn more, see [Quickstart: Use the Azure portal to create a virtual network](https://learn.microsoft.com/en-us/azure/virtual-network/quick-create-portal).
       * **Subnet**: Click **Manage subnet configuration**, or select a new or existing subnet from the dropdown menu. To learn more, see [Add, change, or delete a virtual network subnet](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-subnet?tabs=azure-portal).
       * **Configure network security group**: Click **Create new**, or select an existing security group from the dropdown menu. To learn more, see [Create, change, or delete a network security group](https://learn.microsoft.com/en-us/azure/virtual-network/manage-network-security-group?tabs=network-security-group-portal).

    3. In the **Load balancing** section, fill out the following fields:

       * **Load balancing options**: Select **Azure load balancer**.
       * **Select a load balancer**: Click **Create a load balancer** and fill out the following fields in the pop-up window, or select an existing load balancer from the dropdown menu and note this name as you will need it in later steps:

         * Enter a **Load balancer name**. Note this name, as you will need it in later steps.
         * For **Type**, select **Public** or **Internal**.
         * For **Protocol**, select **TCP** or **UDP**.
         * Set both **Port** and **Backend port** to **80**.

       <img alt="load balancer" />

    4. Click **Create**.
  </Step>

  <Step title="Finalize and deploy">
    1. Click **Review + create**.
    2. Wait for validation.
    3. Click **Create**.

       <img alt="deployment" />
  </Step>

  <Step title="Post-deployment: additional load balancer configuration">
    1. Go to your load balancer: in the Azure portal's **Search resources, services and docs** box, enter and then select **Load balancers**.
    2. Search for and open the new load balancer that you created earlier, or the existing load balancer that you chose earlier.
    3. Make any necessary settings updates to your new or existing load balancer, based on the recommended configurations in the [Load balancer network settings](#load-balancer-network-settings) section, later on this page.
  </Step>

  <Step title="Post-deployment: get the deployed endpoint URL">
    1. Go to your virtual machine: in the Azure portal's **Search resources, services and docs** box, enter and then select **Virtual machines**.

    2. Search for and open the new virtual machine that you created earlier, using the name that you entered earlier.

    3. On the **Overview** tab, under **Properties**, note the **Public IP address** for the **Load balancer**.

       <img alt="retrieve public ip" />

    4. The deployed endpoint URL is **http\://\<load-balancer-public-IP-address>/general/v0/general**. Note this endpoint URL, as you will need it later to call the Unstructured API.
  </Step>

  <Step title="Post-deployment: set API environment variables">
    Note the API environment variables in the [API environment variables](#api-environment-variables) section, later on this page. If you need to set any of these in the Docker container on the virtual machine, do the following:

    1. If the virtual machine is not already running from earlier, click the **Start** icon.

    2. After the virtual machine starts, click the **Connect** icon, and then click **Connect** from the drop-down list.

    3. Follow the on-screen directions for one of the available options to connect to the virtual machine and display a connected terminal.

    4. Stop the running container in the virtual machine, so that you can restart it later with the environment variables set: In the connected terminal, run the following command: `sudo docker container ls`.

    5. Note the `CONTAINER ID` value for the running container.

    6. Run the following command, replacing `<CONTAINER ID>` with the `CONTAINER ID` value:

       ```bash theme={null}
       sudo docker container rm --force <CONTAINER ID>
       ```

    7. Now run the container again, setting the environment variables at the same time: Run the following command: `sudo docker image ls`.

    8. Note the `REPOSITORY` and `TAG` value for the Docker image.

    9. Run the following command, replacing `<REPOSITORY>` and `<TAG>` with the `REPOSITORY` and `TAG` values for the Docker image, and replacing
       `<VAR1>=<value1>`, `<VAR2>=<value2>` and so on with the environment variable name and value pairs:

       ```bash theme={null}
       sudo docker run -d --restart unless-stopped \
       -p 80:5000 \
       -e <VAR1>=<value1> -e <VAR2>=<value2> -e <VAR3>=<value3> \
       <REPOSITORY>:<TAG>
       ```

    10. Verify that the environment variables were set correctly: Run the following command:

        ```bash theme={null}
        sudo docker container ls
        ```

    11. Note the `CONTAINER ID` value for the running container.

    12. Run the following command, replacing `<CONTAINER ID>` with the `CONTAINER ID` value:

    ```bash theme={null}
    sudo docker exec <CONTAINER ID> bash -c 'printenv'
    ```

    14. The environment variables should be in the list that appears.

    <Note>To help manage your overall costs, you should click the **Stop** icon whenever you are not using this virtual machine to call the Unstructured API.</Note>
  </Step>

  <Step title="Call the Unstructured API">
    You can now use the running virtual machine to call the Unstructured API. For example, run one of the following, setting the following environment variables to make your code more portable:

    * Set `UNSTRUCTURED_API_URL` to `http://`, followed by your load balancer's public IP address, followed by `/general/v0/general`.
    * Set `LOCAL_FILE_INPUT_DIR` to the path on your local machine to the files for the Unstructured API to process. If you do not have any input files available, you can download any of the ones from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in GitHub.
    * Set `LOCAL_FILE_OUTPUT_DIR` to the path on your local machine for Unstructured API to send the processed output in JSON format.
  </Step>
</Steps>

<AccordionGroup>
  <Accordion title="Ingest CLI">
    You must first [install the Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the command-line option `--api-key` Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the command-line option `--api-key`.

    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python library">
    You must first [install the Unstructured Ingest Python library](/open-source/ingestion/python-ingest).

    Because you are calling a private API and therefore do not need an Unstructured API key, you can omit the parameter `api_key`. Or, for better code portability, it is recommended that you first set the environment variable `UNSTRUCTURED_API_KEY` to an empty string and then include the parameter `api_key`.

    ```python Python Ingest theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

<Note>To help manage your overall costs, you should stop running the associated virtual machine whenever you are not using it to call the Unstructured API.</Note>

## Load balancer network settings

Unstructured recommends the following load balancer settings, which you should set on your deployment's load balancer soon after you finalize and deploy it.

On the load balancer's **Overview** tab in the Azure portal:

* **SKU**: **Standard**

On the load balancer's **Settings** tab in the Azure portal:

* **Frontend IP configuration**: Private IP

* **Backend pools**: VMSS

* **Health probes**:

  * **Protocol**: **HTTP**, or **HTTPS** (this requires setting up a reverse proxy on the VMSS set to do TLS termination)
  * **Port**: `80` or `443` (this can be any port that the backend VMs are listening on)
  * **Path**: `/healthcheck`
  * **Interval (seconds)**: `5`

* **Load balancing rules**:

  * **Protocol**: **TCP**
  * **Port**: `443` for HTTPS, or `80` for HTTP
  * **Backend port**: `443` for HTTPS, or `80` for HTTP
  * **Idle timeout (minutes)**: `60`
  * **Enable TCP Reset** box: Checked

* **Inbound NAT rules**:

  * **Frontend Port**: `443` for HTTPS, or `80` for HTTP
  * **Backend port**: `443` for HTTPS, or `80` for HTTP
  * **Protocol**: **TCP**
  * **Enable TCP Reset** box: Checked
  * **Idle timeout (minutes)**: `60`

## API environment variables

Unstructured supports the following environment variables, which you can set in the Docker image on the virtual machine, as needed:

* `ALLOW_ORIGINS`: CORS-allowed origins.
* `UNSTRUCTURED_ALLOWED_MIMETYPE`: The list of allowed MIME types, if you want to limit the file types that can be processed.
* `UNSTRUCTURED_API_KEY`: The default Unstructured API key to use.
* `UNSTRUCTURED_MEMORY_FREE_MINIMUM_MB`: The minimum amount of free memory in MB to allow for processing a file. If this memory is too low, the server will return a `503` error.
* `UNSTRUCTURED_PDF_HI_RES_MAX_PAGES`: The maximum number of pages in a PDF file that the Unstructured API will not reject, if the `hi_res` strategy is used. The default is `300`.
* `UNSTRUCTURED_REDIRECT_ROOT_URL`: If this is set, redirect a `GET` request to the Unstructured API to use this URL instead.


# Overview
Source: https://docs.unstructured.io/api-reference/legacy-api/overview



Unstructured has deprecated the following APIs:

* The [Unstructured API on AWS](/api-reference/legacy-api/aws) is deprecated. It is no longer supported and is not being actively updated.
  Unstructured is now available on the AWS Marketplace as a private offering. To explore supported options
  for running Unstructured within your virtual private cloud (VPC), email Unstructured Sales at
  [sales@unstructured.io](mailto:sales@unstructured.io).
* The [Unstructured API on Azure](/api-reference/legacy-api/azure) is deprecated. It is no longer supported and is not being actively updated.
  Unstructured is now available on the AWS Marketplace as a private offering. To explore supported options
  for running Unstructured within your virtual private cloud (VPC), email Unstructured Sales at
  [sales@unstructured.io](mailto:sales@unstructured.io).

Unstructured considers the following APIs to be in a legacy state:

* The [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) is in a legacy state. Although it
  is still supported, Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead.

These pages are not being actively updated. They might contain out-of-date information. These pages are provided for legacy reference purposes only.


# Partition Endpoint parameters
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/api-parameters



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

The Unstructured Partition Endpoint provides parameters to customize the processing of documents. These parameters include:

The only required parameter is `files` -  the file you wish to process.

[//]: # "TODO: In the future we should automatically generate this table to make sure it's up to date"

| POST, Python                               | JavaScript/TypeScript                    | Description                                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `files` (*shared.Files*)                   | `files` (*File*, *Blob*, *shared.Files*) | The file to process.                                                                                                                                                                                                                                                                                                                                   |
| `chunking_strategy` (*str*)                | `chunkingStrategy` (*string*)            | Use one of the supported strategies to chunk the returned elements after partitioning. When no chunking strategy is specified, no chunking is performed and any other chunking parameters provided are ignored. Supported strategies: `basic`, `by_title`, `by_page`, and `by_similarity`. [Learn more](/api-reference/legacy-api/partition/chunking). |
| `content_type` (*str*)                     | `contentType` (*string*)                 | A hint to Unstructured about the content type to use (such as `text/markdown`), when there are problems processing a specific file. This value is a MIME type in the format `type/subtype`. For available MIME types, see [model.py](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/file_utils/model.py).                      |
| `coordinates` (*bool*)                     | `coordinates` (*boolean*)                | True to return bounding box coordinates for each element extracted with OCR. Default: false. [Learn more](/api-reference/legacy-api/partition/examples#saving-bounding-box-coordinates).                                                                                                                                                               |
| `encoding` (*str*)                         | `encoding` (*string*)                    | The encoding method used to decode the text input. Default: `utf-8`.                                                                                                                                                                                                                                                                                   |
| `extract_image_block_types` (*List\[str]*) | `extractImageBlockTypes` (*string\[]*)   | The types of elements to extract, for use in extracting image blocks as Base64 encoded data stored in element metadata fields, for example: `["Image","Table"]`. Supported filetypes are image and PDF. [Learn more](/api-reference/legacy-api/partition/extract-image-block-types).                                                                   |
| `gz_uncompressed_content_type` (*str*)     | `gzUncompressedContentType` (*string*)   | If file is gzipped, use this content type after unzipping. Example: `application/pdf`                                                                                                                                                                                                                                                                  |
| `hi_res_model_name` (*str*)                | `hiResModelName`  (*string*)             | DEPRECATED. The name of the object detection model used when strategy is `hi_res`. Normally, you don't need to specify this, you'll get the best object detection model by default. [Learn more](/api-reference/legacy-api/partition/examples#changing-partition-strategy-for-a-pdf).                                                                  |
| `include_page_breaks` (*bool*)             | `includePageBreaks` (*boolean*)          | True for the output to include page breaks if the filetype supports it. Default: false.                                                                                                                                                                                                                                                                |
| `languages` (*List\[str]*)                 | `languages` (*string\[]*)                | The languages present in the document, for use in partitioning and OCR. [View the list of available languages](https://github.com/tesseract-ocr/tessdata). [Learn more](/api-reference/legacy-api/partition/examples#specifying-the-language-of-a-document-for-better-ocr-results).                                                                    |
| `output_format` (*str*)                    | `outputFormat` (*string*)                | The format of the response. Supported formats are `application/json` and `text/csv`. Default: `application/json`.                                                                                                                                                                                                                                      |
| `pdf_infer_table_structure` (*bool*)       | `pdfInferTableStructure` (*boolean*)     | **Deprecated!** Use `skip_infer_table_types` instead. If true and `strategy` is `hi_res`, any `Table` elements extracted from a PDF will include an additional metadata field, `text_as_html`, where the value (string) is a just a transformation of the data into an HTML table.                                                                     |
| `skip_infer_table_types` (*List\[str]*)    | `skipInferTableTypes` (*string\[]*)      | The document types that you want to skip table extraction for. Default: `[]`.                                                                                                                                                                                                                                                                          |
| `starting_page_number` (*int*)             | `startingPageNumber` (*number*)          | The page number to be be assigned to the first page in the document. This information will be included in elements' metadata and can be be especially useful when partitioning a document that is part of a larger document.                                                                                                                           |
| `strategy` (*str*)                         | `strategy` (*string*)                    | The strategy to use for partitioning PDF and image files. Options are `auto`, `vlm`, `hi_res`, `fast`, and `ocr_only`. Default: `auto`. [Learn more](/api-reference/legacy-api/partition/partitioning).                                                                                                                                                |
| `unique_element_ids` (*bool*)              | `uniqueElementIds` (*boolean*)           | True to assign UUIDs to element IDs, which guarantees their uniqueness (useful when using them as primary keys in database). Otherwise a SHA-256 of the element's text is used. Default: false.                                                                                                                                                        |
| `vlm_model` (*str*)                        | (Not yet available)                      | Applies only when `strategy` is `vlm`. The name of the vision language model (VLM) provider to use for partitioning. `vlm_model_provider` must also be specified. For a list of allowed values, see the end of this article.                                                                                                                           |
| `vlm_model_provider` (*str*)               | (Not yet available)                      | Applies only when `strategy` is `vlm`. The name of the vision language model (VLM) to use for partitioning. `vlm_model` must also be specified. For a list of allowed values, see the end of this article.                                                                                                                                             |
| `xml_keep_tags` (*bool*)                   | `xmlKeepTags` (*boolean*)                | True to retain the XML tags in the output. Otherwise it will just extract the text from within the tags. Only applies to XML documents.                                                                                                                                                                                                                |

The following parameters only apply when a chunking strategy is specified. Otherwise, they are ignored. [Learn more](/api-reference/legacy-api/partition/chunking).

| POST, Python                     | JavaScript/TypeScript             | Description                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `combine_under_n_chars` (*int*)  | `combineUnderNChars` (*number*)   | Applies only when the chunking strategy is set to `by_title`. Use this parameter to combines small chunks until the combined chunk reaches a length of `n` characters. This can mitigate the appearance of small chunks created by short paragraphs, not intended as section headings, being identified as `Title` elements in certain documents. Default: the same value as `max_characters`. |
| `include_orig_elements` (*bool*) | `includeOrigElements` (*boolean*) | True (the default) to have the elements that are used to form a chunk appear in `.metadata.orig_elements` for that chunk.                                                                                                                                                                                                                                                                      |
| `max_characters` (*int*)         | `maxCharacters` (*number*)        | Cut off new sections after reaching a length of `n` characters. (This is a hard maximum.) Default: 500.                                                                                                                                                                                                                                                                                        |
| `multipage_sections` (*bool*)    | `multipageSections` (*boolean*)   | Applies only when the chunking strategy is set to `by_title`. Determines if a chunk can include elements from more than one page. Default: true.                                                                                                                                                                                                                                               |
| `new_after_n_chars` (*int*)      | `newAfterNChars` (*number*)       | Applies only when the chunking strategy is specified. Cuts off new sections after reaching a length of `n` characters. (This is a soft maximum.) Default: 1500.                                                                                                                                                                                                                                |
| `overlap` (*int*)                | `overlap` (*number*)              | A prefix of this many trailing characters from the prior text-split chunk is applied to second and later chunks formed from oversized elements by text-splitting. Default: none.                                                                                                                                                                                                               |
| `overlap_all` (*bool*)           | `overlapAll` (*boolean*)          | True to have an overlap also applied to "normal" chunks formed by combining whole elements. Use with caution, as this can introduce noise into otherwise clean semantic units. Default: none.                                                                                                                                                                                                  |
| `similarity_threshold` (*float*) | `similarityThreshold` (*number*)  | Applies only when the chunking strategy is set to `by_similarity`. The minimum similarity text in consecutive elements must have to be included in the same chunk. Must be between 0.0 and 1.0, exclusive (0.01 to 0.99, inclusive). Default: 0.5.                                                                                                                                             |

The following parameters are specific to the Python and JavaScript/TypeScript clients and are not sent to the server. [Learn more](/api-reference/legacy-api/partition/sdk-python#page-splitting).

| POST, Python                          | JavaScript/TypeScript                 | Description                                                                                                                                                                                                                                          |
| ------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `split_pdf_page` (*bool*)             | `splitPdfPage` (*boolean*)            | True to split the PDF file client-side. [Learn more](/api-reference/legacy-api/partition/sdk-python#page-splitting).                                                                                                                                 |
| `split_pdf_allow_failed` (*bool*)     | `splitPdfAllowFailed` (*boolean*)     | When `true`, a failed split request will not stop the processing of the rest of the document. The affected page range will be ignored in the results. When `false`, a failed split request will cause the entire document to fail. Default: `false`. |
| `split_pdf_concurrency_level` (*int*) | `splitPdfConcurrencyLevel` (*number*) | The number of split files to be sent concurrently. Default: 5. Maximum: 15.                                                                                                                                                                          |
| `split_pdf_page_range` (*List\[int]*) | `splitPdfPageRange` (*number\[]*)     | A list of 2 integers within the range `[1, length_of_pdf]`. When pdf splitting is enabled, this will send only the specified page range to the API.                                                                                                  |

Need help getting started? Check out the [Examples page](/api-reference/legacy-api/partition/examples) for some inspiration.

Allowed values for `vlm_model_provider` and `vlm_model` pairs include the following:

<Warning>
  The following models are no longer available as of the following dates:

  * For `anthropic`, `claude-3-5-sonnet-20241022`: October 22, 2025
  * For `bedrock`, `us.anthropic.claude-3-5-sonnet-20241022-v2:0`: October 22, 2025

  Unstructured recommends the following actions:

  * For new workflows, do not use any of these models.
  * For any workflow that uses any of these models, update that workflow as soon as possible to use a different model.

  Workflows that attempt to use any of these models on or after its associated date will return errors.
</Warning>

| `vlm_model_provider` | `vlm_model`                                  |
| -------------------- | -------------------------------------------- |
| `anthropic`          | `claude-sonnet-4-20250514`                   |
| `bedrock`            | `us.amazon.nova-lite-v1:0`                   |
| `bedrock`            | `us.amazon.nova-pro-v1:0`                    |
| `bedrock`            | `us.anthropic.claude-3-haiku-20240307-v1:0`  |
| `bedrock`            | `us.anthropic.claude-3-opus-20240229-v1:0`   |
| `bedrock`            | `us.anthropic.claude-3-sonnet-20240229-v1:0` |
| `bedrock`            | `us.anthropic.claude-sonnet-4-20250514-v1:0` |
| `bedrock`            | `us.meta.llama3-2-11b-instruct-v1:0`         |
| `bedrock`            | `us.meta.llama3-2-90b-instruct-v1:0`         |
| `openai`             | `gpt-4o`                                     |
| `openai`             | `gpt-5-mini-2025-08-07`                      |
| `vertexai`           | `gemini-2.0-flash-001`                       |


# Endpoint errors
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/api-validation-errors



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

For the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python),
the [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) returns errors primarily through
the `UnstructuredClientError` class (the base class for all errors raised by the Unstructured Python SDK) and
the `HTTPValidationError` class (inherited from `UnstructuredClientError`). Less common errors are returned through the following classes:

* `httpx.RequestError`, the base class for request errors.
* `httpx.ConnectError`, for HTTP connection request errors.
* `httpx.TimeoutException`, for HTTP request timeout errors.
* `ServerError` (inherited from `UnstructuredClientError`), for server-side errors.
* `ResponseValidationError` (inherited from `UnstructuredClientError`), for type mismatches between the response data and the expected Pydantic model.

Each of the preceding classes has the following members:

| Member         | Type             | Description                                                        |
| -------------- | ---------------- | ------------------------------------------------------------------ |
| `message`      | `str`            | The eror message.                                                  |
| `status_code`  | `int`            | The HTTP response status code, for example `401`.                  |
| `headers`      | `httpx.Headers`  | A collection of HTTP response headers.                             |
| `body`         | `str`            | The HTTP body. This can be an empty string if no body is returned. |
| `raw_response` | `httpx.Response` | The raw HTTP response.                                             |

The following example shows how to handle the preceding errors. In this example,
a required Unstructured API key is intentionally commented out of the code, so that a
`401 Unauthorized` error is intentionally thrown.

```python theme={null}
import os
import json

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import PartitionRequest
from unstructured_client.models.shared import (
    PartitionParameters,
    Files,
    Strategy
)
from unstructured_client.models.errors import (
    UnstructuredClientError,
    HTTPValidationError
)
from unstructured_client.models.errors.servererror import ServerError
from unstructured_client.models.errors.responsevalidationerror import ResponseValidationError
import httpx

try:
    client = UnstructuredClient(
        # For example, intentionally leave out the API key to intentionally throw an error.
        # api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    filename = "PATH_TO_INPUT_FILE"

    request = PartitionRequest(
        partition_parameters=PartitionParameters(
            files=Files(
                content=open(filename, "rb"),
                file_name=filename,
            ),
            strategy=Strategy.VLM,
            vlm_model="gpt-4o",
            vlm_model_provider="openai",
            languages=['eng'],
            split_pdf_page=True,            # If True, splits the PDF file into smaller chunks of pages.
            # split_pdf_allow_failed=True,  # If True, the partitioning continues even if some pages fail.
            split_pdf_concurrency_level=15  # Set the number of concurrent request to the maximum value: 15.
        ),
    )

    response = client.general.partition(
        request=request
    )
    element_dicts = [element for element in response.elements]
    
    # Print the processed data's first element only.
    print(element_dicts[0])

    # Write the processed data to a local file.
    json_elements = json.dumps(element_dicts, indent=2)

    with open("PATH_TO_OUTPUT_FILE", "w") as file:
        file.write(json_elements)

except HTTPValidationError as e:
    print("Validation error (HTTP 422):", e)
except ServerError as e:
    print("Server error (HTTP 5XX):", e)
except ResponseValidationError as e:
    print("Response validation/type mismatch:", e)
except UnstructuredClientError as e:
    # This catches any other UnstructuredClientError not already caught above.
    # This and all of the other error classes in this example expose the following members:
    print("Other Unstructured client error:")
    print(f"Message:      {e.message}")
    print(f"Status code:  {e.status_code}")
    print(f"Body:         {e.body}")
    print(f"Raw response: {e.raw_response}")
    print(f"Headers:")

    for header in e.headers.raw:
        key = header[0].decode('utf-8')
        value = header[1].decode('utf-8')
        print(f"              {key}: {value}")

except httpx.ConnectError as e:
    print("HTTP connection error:", e)
except httpx.TimeoutException as e:
    print("HTTP timeout error:", e)
except httpx.RequestError as e:
    # This catches catch-all network errors from HTTP not already caught above.
    print("Other HTTPX request error:", e)
except Exception as e:
    # Optional: this catches any other unforeseen errors.
    print("Unexpected error:", e)
```

The results of running the preceding code are similar to the following:

```text theme={null}
Message:      API error occurred: Status 401. Body: {"detail":"API key is missing, please provide an API key in the header."}
Status code:  401
Body:         {"detail":"API key is missing, please provide an API key in the header."}
Raw response: <Response [401 Unauthorized]>
Headers:
              date: <date-and-time-of-the-error>
              server: <server-identifier>
              content-length: 73
              content-type: application/json
```


# Chunking strategies
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/chunking



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

Chunking functions use metadata and document elements detected with partition functions to split a document into
appropriately-sized chunks for uses cases such as retrieval-augmented generation (RAG).

If you are familiar with chunking methods that split long text documents into smaller chunks, you'll notice that
Unstructured methods slightly differ, since the partitioning step already divides an entire document into its structural elements.

Individual elements will only be split if they exceed the desired maximum chunk size. Two or more consecutive text elements
that will together fit within `max_characters` will be combined. After chunking, you will only have elements of the
following types:

* `CompositeElement`: Any text element will become a `CompositeElement` after chunking. A composite element can be a
  combination of two or more original text elements that together fit within the maximum chunk size. It can also be a single
  element that doesn't leave room in the chunk for any others but fits by itself. Or it can be a fragment of an original
  text element that was too big to fit in one chunk and required splitting.
* `Table`:  A table element is not combined with other elements and if it fits within `max_characters` it will remain as is.
* `TableChunk`: large tables that exceed `max_characters` chunk size are split into special `TableChunk` elements.

### "basic" chunking strategy

* The basic strategy combines sequential elements to maximally fill each chunk while respecting both the specified `max_characters` (hard-max) and `new_after_n_chars` (soft-max) option values.

* A single element that by itself exceeds the hard-max is isolated (never combined with another element) and then divided into two or more chunks using text-splitting.

* A `Table` element is always isolated and never combined with another element. A `Table` can be oversized, like any other text element, and in that case is divided into two or more `TableChunk` elements using text-splitting.

* If specified, `overlap` is applied between chunks formed by splitting oversized elements and is also applied between other chunks when `overlap_all` is `True`.

### "by\_title" chunking strategy

The `by_title` chunking strategy preserves section boundaries and optionally page boundaries as well. “Preserving” here means that a single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one started, even if the next element would fit in the prior chunk.

In addition to the behaviors of the `basic` strategy above, the `by_title` strategy has the following behaviors:

* **Detect section headings.** A `Title` element is considered to start a new section. When a `Title` element is encountered, the prior chunk is closed and a new chunk started, even if the `Title` element would fit in the prior chunk.

* **Respect page boundaries.** Page boundaries can optionally also be respected using the `multipage_sections` argument. This defaults to `True` meaning that a page break does *not* start a new chunk. Setting this to `False` will separate elements that occur on different pages into distinct chunks.

* **Combine small sections.** In certain documents, partitioning may identify a list-item or other short paragraph as a `Title` element even though it does not serve as a section heading. This can produce chunks substantially smaller than desired. This behavior can be mitigated using the `combine_text_under_n_chars` argument. This defaults to the same value as `max_characters` such that sequential small sections are combined to maximally fill the chunking window. Setting this to `0` will disable section combining.

### "by\_page" chunking strategy

Only available in Unstructured Pipelines and the API.

The `by_page` chunking strategy ensures the content from different pages do not end up in the same chunk.
When a new page is detected, the existing chunk is completed and a new one is started, even if the next element would fit in the
prior chunk.

### "by\_similarity" chunking strategy

Only available in Unstructured API and Platform.

The `by_similarity` chunking strategy employs the `sentence-transformers/multi-qa-mpnet-base-dot-v1` embedding model to
identify topically similar sequential elements and combine them into chunks.

As with other strategies, chunks will never exceed the hard-maximum chunk size set by `max_characters`. For this reason,
not all elements that share a topic will necessarily appear in the same chunk. However, with this strategy you can
guarantee that two elements with low similarity will not be combined in a single chunk.

You can control the level of topic similarity you require for elements to have by setting the `similarity_threshold` parameter.
`similarity_threshold` expects a value between 0.0 and 1.0 specifying the minimum similarity text in consecutive elements
must have to be included in the same chunk. The default is 0.5.

###

## Learn more

<Icon icon="blog" />  [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)


# Document elements and metadata
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/document-elements



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

`unstructured` simplifies and streamline the preprocessing of structured and unstructured documents for downstream tasks.
What that means is no matter where your data is and no matter what format that data is in, Unstructured's toolkit will
transform and preprocess that data into an easily digestible and usable format that is uniform across data formats.

When you partition a document with Unstructured, the result is a list of document `Element` objects.
These element objects represent different components of the source document.

## Element example

Here's an example of what an element might look like:

```
{'type': 'NarrativeText',
 'element_id': '5ef1d1117721f0472c1ad825991d7d37',
 'text': 'The Unstructured API documentation covers the following API services:',
 'metadata': {'last_modified': '2024-05-01T14:15:22',
  'page_number': 1,
  'languages': ['eng'],
  'parent_id': '56f24319ae258b735cac3ec2a271b1d9',
  'file_directory': '/content',
  'filename': 'Unstructured - Unstructured.html',
  'filetype': 'text/html'}}
```

As you can see, every element will have a `type`, an `element_id`, the extracted `text`, and some `metadata` which may
vary depending on the element type, document structure, and some additional parameters used during partitioning and/or chunking.

Let's explore some of these document element components in more detail.

## Element type

Instead of treating all documents like a wall of plain text, Unstructured preserves the semantic structure of the documents.
This gives you more control and flexibility over how you further use the processed documents and allows you to take their
structure into consideration. At the same time, normalizing data from various file formats to the Unstructured element
type scheme lets you treat all documents the same in your downstream processing, regardless of source format.
For example, if you plan to summarize a document, you may only be interested in the narrative
of the document, and not care about footers and headers. You can easily filter out the elements you don't need using their type.

Here are some examples of the element types your document may contain:

| Element type        | Description                                                                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Formula`           | An element containing formulas in a document.                                                                                                      |
| `FigureCaption`     | An element for capturing text associated with figure captions.                                                                                     |
| `NarrativeText`     | NarrativeText is an element consisting of multiple, well-formulated sentences. This excludes elements such titles, headers, footers, and captions. |
| `ListItem`          | ListItem is a NarrativeText element that is part of a list.                                                                                        |
| `Title`             | A text element for capturing titles.                                                                                                               |
| `Address`           | A text element for capturing physical addresses.                                                                                                   |
| `EmailAddress`      | A text element for capturing email addresses.                                                                                                      |
| `Image`             | A text element for capturing image metadata.                                                                                                       |
| `PageBreak`         | An element for capturing page breaks.                                                                                                              |
| `Table`             | An element for capturing tables.                                                                                                                   |
| `Header`            | An element for capturing document headers.                                                                                                         |
| `Footer`            | An element for capturing document footers.                                                                                                         |
| `CodeSnippet`       | An element for capturing code snippets.                                                                                                            |
| `PageNumber`        | An element for capturing page numbers.                                                                                                             |
| `UncategorizedText` | Base element for capturing free text from within document.                                                                                         |

If you apply chunking during partitioning of a document or later, you will also see the `CompositeElement` type.
`CompositeElement` is a chunk formed from text (non-Table) elements. It is only produced by chunking.
A composite element may be formed by combining one or more sequential elements produced by partitioning. For example,
several individual list items may be combined into a single chunk.

## Element ID

By default, the element ID is a SHA-256 hash of the element's text, its position on the page, page number it's on,
and the name of the document file - this is to ensure that the ID is deterministic and unique at the document level.
To obtain globally unique IDs in the output (UUIDs), you can pass `unique_element_ids=True` into any of the partition
functions. This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

## Metadata

Unstructured tracks a variety of metadata about the elements extracted from documents.
Here are a couple of examples of what element metadata enables you to do:

* filter document elements based on an element metadata value. For instance, you may want to limit your scope to elements from a certain page, or you may want to use only elements that have an email matching a regular expression in their metadata.
* map an element to the document page where it occurred so that original page can be retrieved when that element matches search criteria.

Metadata is tracked at the element level. You can access the metadata for a given document element with
`element.metadata`. For a dictionary representation, use `element.metadata.to_dict()`.

### Common metadata fields

All document types return the following metadata fields when the information is available from the source file:

| Metadata field name        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename`                 | Filename                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `file_directory`           | File directory                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `last_modified`            | Last modified Date                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `filetype`                 | File type                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `coordinates`              | XY Bounding Box Coordinates. See notes below for further details about the bounding box.                                                                                                                                                                                                                                                                                                                                                                                           |
| `parent_id`                | Element Hierarchy. `parent_id` may be used to infer where an element resides within the overall hierarchy of a document. For instance, a NarrativeText element may have a Title element as a parent (a “sub-title”), which in turn may have another Title element as its parent (a "title").                                                                                                                                                                                       |
| `category_depth`           | Element depth relative to other elements of the same category.  Category depth is the depth of an element relative to other elements of the same category. It’s set by a document partitioner and enables the hierarchy post-processor to compute more accurate hierarchies. Category depth may be set using native document hierarchies, e.g. reflecting \<H1>, \<H2>, or \<H3> tags within an HTML document or the indentation level of a bulleted list item in a Word document. |
| `image_base64`             | A Base64 representation of the detected image or table. Only applicable to image and table elements when High Res partitioning is used. After chunking, `image_base64` is not preserved in the output.                                                                                                                                                                                                                                                                             |
| `image_mime_type`          | MIME type of the image. Only applicable to image elements.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `text_as_html`             | HTML representation of extracted tables. Only applicable to table elements.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `languages`                | Document Languages.  At document level or element level. List is ordered by probability of being the primary language of the text.                                                                                                                                                                                                                                                                                                                                                 |
| `emphasized_text_contents` | Emphasized text (bold or italic) in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `emphasized_text_tags`     | Tags on text that is emphasized in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `orig_elements`            | For chunked elements, a list of the original elements that were used to create the current chunked element.                                                                                                                                                                                                                                                                                                                                                                        |
| `is_continuation`          | True if element is a continuation of a previous element.  Only relevant for chunking, if an element was divided into two due to max\_characters.                                                                                                                                                                                                                                                                                                                                   |
| `detection_class_prob`     | Detection model class probabilities.  From unstructured-inference, hi-res strategy.                                                                                                                                                                                                                                                                                                                                                                                                |

Notes on common metadata fields:

#### Metadata for document hierarchy

`parent_id` and `category_depth` enhance hierarchy detection to identify the document
structure in various file formats by measuring relative depth of an element within its category. This is especially
useful in documents with native hierarchies like HTML or Word files, where elements like headings or list items inherently define structure.

#### Element's coordinates

Some document types support location data for the elements, usually in the form of bounding boxes.
If it exists, an element's location data is available with `element.metadata.coordinates`. The `coordinates` property of an `ElementMetadata` stores:

* `points` : These specify the corners of the bounding box starting from the top left corner and proceeding counter-clockwise. The points represent pixels, the origin is in the top left and the `y` coordinate increases in the downward direction.
* `system`: The points have an associated coordinate system. A typical example of a coordinate system is `PixelSpace`, which is used for representing the coordinates of images. The coordinate system has a name, orientation, layout width, and layout height.

The Unstructured Open Source library offers a way to change the coordinates of an element to a new coordinate system by
using the `Element.convert_coordinates_to_new_system` method. If the `in_place` flag is `True`, the coordinate system
and points of the element are updated in place and the new coordinates are returned. If the `in_place` flag is `False`,
only the altered coordinates are returned.

```python theme={null}
from unstructured.documents.elements import Element
from unstructured.documents.coordinates import PixelSpace, RelativeCoordinateSystem

coordinates = ((10, 10), (10, 100), (200, 100), (200, 10))
coordinate_system = PixelSpace(width=850, height=1100)

element = Element(coordinates=coordinates, coordinate_system=coordinate_system)
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

element.convert_coordinates_to_new_system(RelativeCoordinateSystem(), in_place=True)
# Should now be in terms of new coordinate system
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

```

### Additional metadata fields by document type

| Field name             | Applicable file types | Description                                                                                                                                                          |
| ---------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `attached_to_filename` | MSG                   | The name of the file that the attached file is attached to.                                                                                                          |
| `bcc_recipient`        | EML                   | The related [email](#email) BCC recipient.                                                                                                                           |
| `cc_recipient`         | EML                   | The related [email](#email) CC recipient.                                                                                                                            |
| `email_message_id`     | EML                   | The related [email](#email) message ID.                                                                                                                              |
| `header_footer_type`   | Word Doc              | The pages that a header or footer applies to in a [Word document](#microsoft-word-files): `primary`, `even_only`, and `first_page`.                                  |
| `image_path`           | PDF                   | The path to the image. This is useful when you want to extract the image and save it in a specified path instead of serializing the image within the processed data. |
| `image_mime_type`      | PDF                   | The MIME type of the image.                                                                                                                                          |
| `image_url`            | HTML                  | The URL to the image.                                                                                                                                                |
| `link_start_indexes`   | HTML, PDF             | A list of the index locations within the extracted content where the `links` can be found.                                                                           |
| `link_texts`           | HTML                  | A list of text strings that are associated with the `link_urls`.                                                                                                     |
| `link_urls`            | HTML                  | A list of URLs within the extracted content.                                                                                                                         |
| `links`                | PDF                   | A list of links within the extracted content.                                                                                                                        |
| `page_name`            | XLSX                  | The related sheet's name in an [Excel file](#microsoft-excel-files).                                                                                                 |
| `page_number`          | DOCX, PDF, PPT, XLSX  | The related file's page number.                                                                                                                                      |
| `section`              | EPUB                  | The book section title corresponding to a table of contents.                                                                                                         |
| `sent_from`            | EML                   | The related [email](#email) sender.                                                                                                                                  |
| `sent_to`              | EML                   | The related [email](#email) recipient.                                                                                                                               |
| `signature`            | EML                   | The related [email](#email) signature.                                                                                                                               |
| `subject`              | EML                   | The related [email](#email) subject.                                                                                                                                 |

Notes on additional metadata by document type:

#### Email

For emails, metadata will contain the following fields, where available:

* `bcc_recipient`
* `cc_recipient`
* `email_message_id`
* `sent_from`
* `sent_to`
* `signature`
* `subject`

`sent_from` is a list of strings because the [RFC 822](https://www.rfc-editor.org/rfc/rfc822) spec for emails allows for multiple sent from email addresses.

#### Microsoft Excel documents

For Excel documents, `ElementMetadata` will contain a `page_name` element, which corresponds to the sheet name in the Excel
document.

#### Microsoft Word documents

Headers and footers in Word documents include a `header_footer_type` indicating which page a header or footer applies to.
Valid values are `"primary"`, `"even_only"`, and `"first_page"`.

### Table-specific metadata

For `Table` elements, the raw text of the table will be stored in the `text` attribute for the Element, and HTML representation
of the table will be available in the element metadata under `element.metadata.text_as_html`. By default,
Unstructured will automatically extract all tables for all doc types unless you set `skip_infer_table_types` parameter.

Here's an example of a table element. The `text` of the element will look like this:

```
Dataset Base Model1 Large Model Notes PubLayNet [38] F / M M Layouts of modern scientific documents PRImA [3] M - Layouts of scanned modern magazines and scientific reports Newspaper [17] F - Layouts of scanned US newspapers from the 20th century TableBank [18] F F Table region on modern scientific and business document HJDataset [31] F / M - Layouts of history Japanese documents
```

And the `text_as_html` metadata for the same element will look like this:

```py theme={null}
<table><thead><th>Dataset</th><th>| Base Model’</th><th>| Notes</th></thead><tr><td>PubLayNet</td><td>[38] F/M</td><td>Layouts of modern scientific documents</td></tr><tr><td>PRImA [3]</td><td>M</td><td>Layouts of scanned modern magazines and scientific reports</td></tr><tr><td>Newspaper</td><td>F</td><td>Layouts of scanned US newspapers from the 20th century</td></tr><tr><td>TableBank</td><td>F</td><td>Table region on modern scientific and business document</td></tr><tr><td>HJDataset [31]</td><td>F/M</td><td>Layouts of history Japanese documents</td></tr></table>
```

### Data connector metadata fields

Documents processed through source connectors include additional document metadata. These additional fields only ever
appear if the source document was processed by a connector.

#### Common data connector metadata fields

* Data Source metadata (on json output):
  * url
  * version
  * date created
  * date modified
  * date processed
  * record locator
* Record locator is specific to each connector

#### Additional metadata fields by connector type (via record locator)

| Source connector      | Additional metadata              |
| --------------------- | -------------------------------- |
| airtable              | base id, table id, view id       |
| azure (from fsspec)   | protocol, remote file path       |
| box (from fsspec)     | protocol, remote file path       |
| confluence            | url, page id                     |
| discord               | channel                          |
| dropbox (from fsspec) | protocol, remote file path       |
| elasticsearch         | url, index name, document id     |
| fsspec                | protocol, remote file path       |
| google drive          | drive id, file id                |
| gcs (from fsspec)     | protocol, remote file path       |
| jira                  | base url, issue key              |
| onedrive              | user pname, server relative path |
| outlook               | message id, user email           |
| s3 (from fsspec)      | protocol, remote file path       |
| sharepoint            | server path, site url            |
| wikipedia             | page title, age url              |


# Examples
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/examples

This page provides some examples of accessing Unstructured Partition Endpoint via different methods.

<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

To use these examples, you'll first need to set an environment variable named `UNSTRUCTURED_API_KEY`,
representing your Unstructured API key. [Get your API key](/api-reference/legacy-api/partition/overview).

Also, you'll need to set an environment variable named `UNSTRUCTURED_API_URL` to the
value of the Unstructured API URL for your account. This API URL was provided to you when your Unstructured account was created.
If you do not have this API URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  For the Unstructured Python SDK, the API URL is set to `https://api.unstructuredapp.io/general/v0/general` by default.
  However, you should always use the API URL that was provided to you when your Unstructured account was created.
</Note>

### Changing partition strategy for a PDF

Here's how you can modify partition strategy for a PDF file, and select an alternative model to use with Unstructured API.

<AccordionGroup>
  <Accordion title="POST">
    <UseIngestOrPlatformInstead />

    ```bash POST theme={null}
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper.pdf' \
    -F 'strategy=vlm' \
    -F 'vlm_model_provider=openai' \
    -F 'vlm_model=gpt-4o'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <UseIngestOrPlatformInstead />

    ```python Python theme={null}
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.VLM,
                "vlm_model": "gpt-4o",
                "vlm_model_provider": "openai",
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(
                request=req
            )
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <UseIngestOrPlatformInstead />

    ```typescript TypeScript theme={null}
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.HiRes,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

If you have a local deployment of the Unstructured API, you can use other supported models, such as `yolox`.

### Specifying the language of a document for better OCR results

For better OCR results, you can specify what languages your document is in using the `languages` parameter.
[View the list of available languages](https://github.com/tesseract-ocr/tessdata).

<AccordionGroup>
  <Accordion title="POST">
    <UseIngestOrPlatformInstead />

    ```bash POST theme={null}
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/korean.png' \
    -F 'strategy=vlm' \
    -F 'vlm_model_provider=openai' \
    -F 'vlm_model=gpt-4o' \-F 'languages=kor'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <UseIngestOrPlatformInstead />

    ```python Python theme={null}
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.OCR_ONLY,
                "languages": ["kor"],
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(
                request=req
            )
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <UseIngestOrPlatformInstead />

    ```typescript TypeScript theme={null}
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.OcrOnly,
                        languages: ["kor"],
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

### Saving bounding box coordinates

When elements are extracted from PDFs or images, it may be useful to get their bounding boxes as well.
Set the `coordinates` parameter to `true` to add this field to the elements in the response.

<AccordionGroup>
  <Accordion title="POST">
    <UseIngestOrPlatformInstead />

    ```bash POST theme={null}
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper.pdf' \
    -F 'coordinates=true' \
    -F 'strategy=hi_res'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <UseIngestOrPlatformInstead />

    ```python Python theme={null}
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.HI_RES,
                "coordinates": True,
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(
                request=req
            )
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <UseIngestOrPlatformInstead />

    ```typescript TypeScript theme={null}
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.HiRes,
                        coordinates: true,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

### Returning unique element IDs

By default, the element ID is a SHA-256 hash of the element text. This is to ensure that
the ID is deterministic. One downside is that the ID is not guaranteed to be unique.
Different elements with the same text will have the same ID, and there could also be hash collisions.
To use UUIDs in the output instead, set `unique_element_ids=true`. Note: this means that the element IDs
will be random, so with every partition of the same file, you will get different IDs.
This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

<AccordionGroup>
  <Accordion title="POST">
    <UseIngestOrPlatformInstead />

    ```bash POST theme={null}
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json'  \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \
    -F 'unique_element_ids=true' \
    -F 'strategy=vlm' \
    -F 'vlm_model_provider=openai' \
    -F 'vlm_model=gpt-4o'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <UseIngestOrPlatformInstead />

    ```python Python theme={null}
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "strategy": shared.Strategy.VLM,
                "vlm_model": "gpt-4o",
                "vlm_model_provider": "openai",
                "unique_element_ids": True,
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(
                request=req
            )
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <UseIngestOrPlatformInstead />

    ```typescript TypeScript theme={null}
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        uniqueElementIds: true,
                        strategy: Strategy.HiRes,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>

### Adding the chunking step after partitioning

You can combine partitioning and subsequent chunking in a single request by setting the `chunking_strategy` parameter.
By default, the `chunking_strategy` is set to `None`, and no chunking is performed.

[//]: # "TODO: add a link to the concepts section about chunking strategies. Need to create the shared Concepts section first"

<AccordionGroup>
  <Accordion title="POST">
    <UseIngestOrPlatformInstead />

    ```bash POST theme={null}
    curl -X 'POST' $UNSTRUCTURED_API_URL \
    -H 'accept: application/json'  \
    -H 'Content-Type: multipart/form-data' \
    -H 'unstructured-api-key: $UNSTRUCTURED_API_KEY' \
    -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \
    -F 'chunking_strategy=by_title' \
    -F 'max_characters=1024' \
    -F 'strategy=vlm' \
    -F 'vlm_model_provider=openai' \
    -F 'vlm_model=gpt-4o'
    ```
  </Accordion>

  <Accordion title="Python SDK">
    <UseIngestOrPlatformInstead />

    ```python Python theme={null}
    import asyncio
    import os
    import json
    import unstructured_client
    from unstructured_client.models import shared

    client = unstructured_client.UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    async def call_api(filename, input_dir, output_dir):
        req = {
            "partition_parameters": {
                "files": {
                    "content": open(filename, "rb"),
                    "file_name": os.path.basename(filename),
                },
                "chunking_strategy": "by_title",
                "max_characters": 1024,
                "strategy": shared.Strategy.VLM,
                "vlm_model": "gpt-4o",
                "vlm_model_provider": "openai",
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        }

        try:
            res = await client.general.partition_async(
                request=req
            )
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)

            # Create the output directory structure.
            relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
            output_subdir = os.path.join(output_dir, relative_path)
            os.makedirs(output_subdir, exist_ok=True)

            # Write the output file.
            output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
            with open(output_filename, "w") as file:
                file.write(json_elements)

        except Exception as e:
            print(f"Error processing {filename}: {e}")

    async def process_files(input_directory, output_directory):
        tasks = []

        for root, _, files in os.walk(input_directory):
            for file in files:
                if not file.endswith('.json'):
                    full_path = os.path.join(root, file)
                    tasks.append(call_api(full_path, input_directory, output_directory))

        await asyncio.gather(*tasks)

    if __name__ == "__main__":
        asyncio.run(process_files(
            input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
            output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
        ))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <UseIngestOrPlatformInstead />

    ```typescript TypeScript theme={null}
    import { UnstructuredClient } from "unstructured-client";
    import * as fs from "fs";
    import * as path from "path";
    import { ChunkingStrategy, Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    // Send all files in the source path to Unstructured for processing.
    // Send the processed data to the destination path.
    function processFiles(
        client: UnstructuredClient,
        sourcePath: string,
        destinationPath: string
    ): void {

        // If an output directory does not exist for the corresponding input
        // directory, then create it.
        if (!fs.existsSync(destinationPath)) {
            fs.mkdirSync(destinationPath, { recursive: true });
        }

        // Get all folders and files at the current level of the input directory.
        const items = fs.readdirSync(sourcePath);

        // For each folder and file in the input directory...
        for (const item of items) {
            const inputPath = path.join(sourcePath, item);
            const outputPath = path.join(destinationPath, item)

            // If it's a folder, call this function recursively.
            if (fs.statSync(inputPath).isDirectory()) {
                processFiles(client, inputPath, outputPath);
            } else {
                // If it's a file, send it to Unstructured for processing.
                const data = fs.readFileSync(inputPath);

                client.general.partition({
                    partitionParameters: {
                        files: {
                            content: data,
                            fileName: inputPath
                        },
                        strategy: Strategy.HiRes,
                        chunkingStrategy: ChunkingStrategy.ByTitle,
                        maxCharacters: 1024,
                        splitPdfPage: true,
                        splitPdfConcurrencyLevel: 15,
                        splitPdfAllowFailed: true
                    }
                }).then((res: PartitionResponse) => {
                    // If successfully processed, write the processed data to
                    // the destination directory.
                    if (res.statusCode == 200) {
                        const jsonElements = JSON.stringify(res, null, 2)
                        fs.writeFileSync(outputPath + ".json", jsonElements)
                    }
                }).catch((e) => {
                    if (e.statusCode) {
                        console.log(e.statusCode);
                        console.log(e.body);
                    } else {
                        console.log(e);
                    }
                });
            }
        }
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
        serverURL: process.env.UNSTRUCTURED_API_URL
    });

    processFiles(
        client,
        process.env.LOCAL_FILE_INPUT_DIR,
        process.env.LOCAL_FILE_OUTPUT_DIR
    );
    ```
  </Accordion>
</AccordionGroup>


# Extract images and tables from documents
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/extract-image-block-types



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

## Task

You want to get, decode, and show elements, such as images and tables, that are embedded in a PDF document.

## Approach

Extract the Base64-encoded representation of specific elements, such as images and tables, in the document.
For each of these extracted elements, decode the Base64-encoded representation of the element into its original visual representation
and then show it.

## To run this example

You will need a document that is one of the document types supported by the `extract_image_block_types` argument.
See the `extract_image_block_types` entry in [API Parameters](/api-reference/legacy-api/partition/api-parameters).
This example uses a PDF file with embedded images and tables.

## Code

For the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python), you'll need:

These environment variables:

* `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
* `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

```python Python SDK theme={null}
from unstructured_client import UnstructuredClient
from unstructured_client.models import operations, shared
from unstructured.staging.base import elements_from_dicts, elements_to_json

import os
import base64
from PIL import Image
import io

if __name__ == "__main__":
    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    # Source: https://github.com/Unstructured-IO/unstructured/blob/main/example-docs/embedded-images-tables.pdf
    
    # Where to get the input file and store the processed data, relative to this .py file.
    local_input_filepath = "local-ingest-input-pdf/embedded-images-tables.pdf"
    local_output_filepath = "local-ingest-output/embedded-images-tables.json"

    with open(local_input_filepath, "rb") as f:
        files = shared.Files(
            content=f.read(),
            file_name=local_input_filepath
        )

    request = operations.PartitionRequest(
        shared.PartitionParameters(
            files=files,
            split_pdf_page=True,
            split_pdf_allow_failed=True,
            split_pdf_concurrency_level=15,
            # Extract the Base64-encoded representation of each
            # processed "Image" and "Table" element. Extract each into
            # an "image_base64" object, as a child of the
            # "metadata" object, for that element in the result.
            # Element type names, such as "Image" and "Table" here,
            # are case-insensitive.
            # Any available Unstructured element type is allowed.
            extract_image_block_types=["Image", "Table"]
        )
    )

    try:
        result = client.general.partition(
            request=request
        )

        for element in result.elements:
            if "image_base64" in element["metadata"]:
                # Decode the Base64-encoded representation of the 
                # processed "Image" or "Table" element into its original
                # visual representation, and then show it.
                image_data = base64.b64decode(element["metadata"]["image_base64"])
                image = Image.open(io.BytesIO(image_data))
                image.show()
        
        # Optionally, prepare to print or save the elements as JSON.
        dict_elements = elements_from_dicts(
            element_dicts=result.elements
        )

        # Print the elements as JSON...
        json_elements = elements_to_json(
            elements=dict_elements,
            indent=2
        )

        print(json_elements)

        # ...or save as JSON.
        elements_to_json(
            elements=dict_elements,
            indent=2,
            filename=local_output_filepath
        )
    except Exception as e:
        print(e)
```

## See also

* [Extract text as HTML](/api-reference/legacy-api/partition/text-as-html)
* [Table extraction from PDF](/examplecode/codesamples/apioss/table-extraction-from-pdf)


# Generate a JSON schema for a file
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/generate-schema



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

## Task

You want to generate a schema for a JSON file that Unstructured produces, so that you can validate, test,
and document related JSON files across your systems.

## Approach

Use a Python package such as [genson](https://pypi.org/project/genson/) to generate schemas for your
JSON files.

<Info>The `genson` package is not owned or supported by Unstructured. For questions and
requests, see the [Issues](https://github.com/wolverdude/genson/issues) tab of the
`genson` repository in GitHub.</Info>

## Generate a schema from the terminal

<Steps>
  <Step title="Install genson">
    Use [pip](https://pip.pypa.io/en/stable/installation/) to install the [genson](https://pypi.org/project/genson/) package.

    ```bash theme={null}
    pip install genson
    ```
  </Step>

  <Step title="Install jq">
    By default, `genson` generates the JSON schema as a single string without any line breaks or indented whitespace.

    To pretty-print the schema that `genson` produces, install the [jq](https://jqlang.github.io/jq/) utility.

    <Info>The `jq` utility is not owned or supported by Unstructured. For questions and
    requests, see the [Issues](https://github.com/jqlang/jq/issues) tab of the
    `jq` repository in GitHub.</Info>
  </Step>

  <Step title="Generate the schema">
    1. Run the `genson` command, specifying the path to the input (source) JSON file, and the path to
       the output (target) JSON schema file to be generated. Use `jq` to pretty-print the schema's content
       into the file to be generated.

       ```bash theme={null}
       genson "/path/to/input/file.json" | jq '.' > "/path/to/output/schema.json"
       ```

    2. You can find the generated JSON schema file in the output path that you specified.
  </Step>
</Steps>

## Generate a schema from Python code

<Steps>
  <Step title="Install dependencies">
    In your Python project, install the [genson](https://pypi.org/project/genson/) package.

    ```bash theme={null}
    pip install genson
    ```
  </Step>

  <Step title="Add and run the schema generation code">
    1. Set the following local environment variables:

       * Set `LOCAL_FILE_INPUT_PATH` to the local path to the input (source) JSON file.
       * Set `LOCAL_FILE_OUTPUT_PATH` to the local path to the output (target) JSON schema file to be generated.

    2. Add the following Python code file to your project:

       ```python theme={null}
       import os, json
       from genson import SchemaBuilder

       def json_schema_from_file(
           input_file_path: str,
           output_schema_path: str
       ) -> None:
           try:
               with open(input_file_path, "r") as file:
                   json_data = json.load(file)

               builder = SchemaBuilder()
               builder.add_object(json_data)

               schema = builder.to_schema()

               try:
                   with open(output_schema_path, "w") as schema_file:
                       json.dump(schema, schema_file, indent=2)
               except IOError as e:
                   raise IOError(f"Error writing to output file: {e}")
               
               print(f"JSON schema successfully generated and saved to '{output_schema_path}'.")
           except FileNotFoundError:
               print(f"Error: Input file '{input_file_path}' not found.")
           except IOError as e:
               print(f"I/O error occurred: {e}")
           except Exception as e:
               print(f"An unexpected error occurred: {e}")

       if __name__ == "__main__":
           json_schema_from_file(
               input_file_path=os.getenv("LOCAL_FILE_INPUT_PATH"),
               output_schema_path=os.getenv("LOCAL_FILE_OUTPUT_PATH")
           )
       ```

    3. Run the Python code file.

    4. Check the path specified by `LOCAL_FILE_OUTPUT_PATH` for the generated JSON schema file.
  </Step>
</Steps>


# Get chunked elements
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/get-chunked-elements



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

## Task

You want to get, and print or save, the contents of elements that have been chunked.

## Approach

Chunked elements are typically represented in this format:

```json theme={null}
{
  "type": "CompositeElement",
  "element_id": "3800f7ff047e97cbf0a5f7df7ad52c80",
  "text": "4,5 Exception noted in \u201cSpecial Considerations for Promotional Labeling and Advertising Material\u201d\n\n3",
  "metadata": {
    "filetype": "application/pdf",
    "languages": ["eng"],
    "page_number": 6,
    "orig_elements": "eJy ... Full content omitted for brevity ... x8=",
    "filename": "Portable-Document-Format-Specifications.pdf"
  }
}
```

```json theme={null}
{
  "type": "Table",
  "element_id": "758127b42c51b93b59abf08640d1ccab",
  "text": "Gels and karyotypes High pressure liquid chromatography 600 dpi (8 bit grayscale depth) 300 dpi",
  "metadata": {
    "text_as_html": "<table><tbody><tr><td>Gels and karyotypes</td><td>600 dpi (8 bit grayscale depth)</td></tr><tr><td>High pressure liquid chromatography</td><td>300</td></tr></tbody></table>",
    "filetype": "application/pdf",
    "languages": ["eng"],
    "page_number": 8,
    "orig_elements": "eJy ... Full content omitted for brevity ... MnD",
    "filename": "Portable-Document-Format-Specifications.pdf"
}
```

To get the element's chunked content, extract the contents of the element's `text` field.

The chunked content might not contain all of its associated content. To get all of the elements that were used to derive this chunked content, extract the contents of the element's `orig_elements` field, which is nested inside of its parent `metadata` field.

The contents of the `orig_elements` field is in compressed Base64 gzipped format. To get the content, Base64-decode the bytes, decompress them, and then decode them using UTF-8.

## To run this example

You will need to chunk a document during processing. This example uses a PDF file chunked into 200- to 300-character elements.

## Code

For the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python), you'll need:

These environment variables:

* `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
* `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

```python theme={null}
from unstructured_client import UnstructuredClient
from unstructured_client.models import operations, shared
from unstructured.staging.base import elements_from_dicts, elements_to_json

import os, base64, zlib
from typing import List, Dict, Any

# Extract the contents of an orig_elements field.
def extract_orig_elements(orig_elements):
    decoded_orig_elements = base64.b64decode(orig_elements)
    decompressed_orig_elements = zlib.decompress(decoded_orig_elements)
    return decompressed_orig_elements.decode('utf-8')

# Source file: https://www.fda.gov/files/drugs/published/Portable-Document-Format-Specifications.pdf
input_filepath = "local-ingest-input-pdf/Portable-Document-Format-Specifications.pdf"
output_filepath = "local-ingest-output-json/Portable-Document-Format-Specifications.json"

client = UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
)

with open(input_filepath, "rb") as f:
    files = shared.Files(
        content=f.read(),
        file_name=input_filepath
    )

# Chunk the document with a basic chunking strategy.
# Create chunks that are at least around 200 characters (soft limit)
# but never more than 300 characters (hard maximum).
req = operations.PartitionRequest(
    shared.PartitionParameters(
        files=files,
        strategy=shared.Strategy.VLM,
        vlm_model="gpt-4o",
        vlm_model_provider="openai",
        split_pdf_page=True,
        split_pdf_allow_failed=True,
        split_pdf_concurrency_level=15,
        chunking_strategy="basic",
        new_after_n_chars=200,
        max_characters=300
    )
)

try:
    res = await client.general.partition_async(
        request=req
    )

    # Create a dictionary that will hold only
    # a transposed version of the returned elements. 
    # For instance, we just want to capture each element's ID,
    # the chunk's text, and the chunk's associated elements in context.
    orig_elements_dict: List[Dict[str, Any]] = []

    for element in res.elements:
        # For each chunk that has an "orig_elements" field...
        if "orig_elements" in element["metadata"]:
            # ...get the chunk's associated elements in context...
            orig_elements = extract_orig_elements(element["metadata"]["orig_elements"])
            # ...and then transpose it and other associated fields into a separate dictionary.
            orig_elements_dict.append({
                "element_id": element["element_id"],
                "text": element["text"],
                "orig_elements": json.loads(orig_elements)
            })
    
    # Convert the elements into a JSON object.
    orig_elements_json = json.dumps(orig_elements_dict, indent=2)

    # Write the JSON to a file.
    with open(output_filepath, "w") as file:
        file.write(orig_elements_json)
except Exception as e:
    print(e)
```

## See also

* [Recovering chunk elements](/open-source/core-functionality/chunking#recovering-chunk-elements)
* [Chunking strategies](/api-reference/legacy-api/partition/chunking)


# Get element contents
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/get-elements



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

## Task

You want to get, manipulate, and print or save, the contents of the [document elements and metadata](/api-reference/legacy-api/partition/document-elements) from the processed data that Unstructured returns.

## Approach

Each element in the document elements contains fields for that element's type, its ID, the extracted text, and associated metadata.

The programmatic approach you take to get these document elements will depend on which SDK you use:

<AccordionGroup>
  <Accordion title="Python SDK">
    For the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python), calling an `UnstructuredClient` object's `general.partition_async` method returns a `PartitionResponse` object.

    This `PartitionResponse` object's `elements` variable contains a list of key-value dictionaries (`List[Dict[str, Any]]`). For example:

    ```python Python theme={null}
    # ...

    res = await client.general.partition_async(request=req)

    # Do something with the elements, for example:
    save_elements_to_file(res.elements)

    # ...
    ```

    You can use standard Python [list operations](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists) on this list.

    You can also use standard Python [looping techniques](https://docs.python.org/3/tutorial/datastructures.html#looping-techniques) on this list to access each element in this list.

    To work with an individual element's contents, you can use standard [dictionary operations](https://docs.python.org/3/library/stdtypes.html#dict) on the element.

    For example:

    ```python Python theme={null}
    # ...

    res = await client.general.partition_async(request=req)

    for element in res.elements:
        # Do something with each element, for example:
        save_element_to_database(f"{element["element_id"]}")
        save_element_to_database(f"{element["text"]}")
        save_element_to_database(f"{element["metadata"]["filename"]}\n")

    # ...
    ```

    To serialize this list as JSON, you can:

    1. Use the `elements_from_dicts` function to convert the list of key-value dictionaries (`Iterable[Dict[str, Any]]`) into a list of elements (`Iterable[Element]`).
    2. Use the `elements_to_json` function to convert the list of elements into a JSON-formatted string and then print or save that string.

    For example:

    ```python Python theme={null}
    from unstructured.staging.base import elements_from_dicts, elements_to_json

    # ...

    res = await client.general.partition_async(request=req)

    dict_elements = elements_from_dicts(
        element_dicts=res.elements
    )

    elements_to_json(
        elements=dict_elements,
        indent=2,
        filename=output_filepath
    )
    # ...
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    For the [Unstructured JavaScript/TypeScript SDK](/api-reference/legacy-api/partition/sdk-jsts), calling an `UnstructuredClient` object's `general.partition` method returns a `Promise<PartitionResponse>` object.

    This `PartitionResponse` object's `elements` property contains an `Array` of string-value objects (`{ [k: string]: any; }[]`). For example:

    ```typescript TypeScript theme={null}
    // ...

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then((res) => {
        if (res.statusCode == 200) {
            // Do something with the elements, for example:
            saveElementsToFile(res)
        }
    } // ...
    ```

    You can use standard [Array operations](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array) on this array.

    You can also use standard `Array` techniques such as [forEach](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach) to access each object in this array. For example:

    ```typescript TypeScript theme={null}
    // ...

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then((res) => {
        if (res.statusCode == 200) {
            res.forEach(element => {
                // Do something with each element, for example:
                saveElementToDatabase(`${element["element_id"]}`)
                saveElementToDatabase(`${element["text"]}`)
                saveElementToDatabase(`${element["metadata"]["filename"]}`)
            }
        }
    } // ...
    ```

    To serialize this list as JSON, you can use the standard [JSON.stringify](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify) function to serialize it to JSON-formatted string and the Node.js [fs.WriteFileSync](https://nodejs.org/api/fs.html#fswritefilesyncfile-data-options) function to save it as a file. For example:

    ```typescript TypeScript theme={null}
    // ...

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then((res) => {
        if (res.statusCode == 200) {    
            const jsonElements = JSON.stringify(res, null, 2)

            fs.writeFileSync(outputFilepath, jsonElements)
        }
    } // ...
    ```
  </Accordion>
</AccordionGroup>


# Overview
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/overview



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

<Tip>
  To start using the Unstructured Partition Endpoint right away, skip ahead to the [quickstart](/api-reference/legacy-api/partition/quickstart) now!
</Tip>

The Unstructured Partition Endpoint, part of the [Unstructured API](/api-reference/overview), is intended for rapid prototyping of Unstructured's
various partitioning strategies, with limited support for chunking. It is designed to work only with processing of local files, one file
at a time. Use the [Unstructured API's workflow operations](/api-reference/workflow/overview) for production-level scenarios, file processing in
batches, files and data in remote locations, generating embeddings, applying post-transform enrichments, using the latest and
highest-performing models, and for the highest quality results at the lowest cost.

## Get started

To call the Unstructured Partition Endpoint, you need an Unstructured account and an Unstructured API key:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

[Try the quickstart](#quickstart).

## Pricing

Unstructured offers different account types with different pricing plans:

* <Icon icon="person" />  **Let's Go** and **Pay-As-You-Go** - A single user, with a single workspace, hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="building" />  **Business** - Multiple users and workspaces, with three options:

  * <Icon icon="people-group" />  **Business SaaS** - Hosted alongside other accounts on Unstructured's cloud infrastructure.
  * <Icon icon="shield-halved" />  **Dedicated instance** - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control.
  * <Icon icon="shield" />  **In-VPC** - Hosted within your own VPC on your own cloud infrastructure.

  **Business** accounts also allow for robust customization of Unstructured's features for your unique needs.

For more details, see the [Unstructured Pricing](https://unstructured.io/pricing) page.

To upgrade your account from **Let's Go** or **Pay-As-You-Go** to **Business**,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Some of these plans have billing details that are determined on a per-page basis.

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

## Quickstart

<Tip>
  To use the Unstructured Python SDK instead of `curl` for the following quickstart,
  [skip ahead](/api-reference/legacy-api/partition/quickstart) to the SDK version instead.
</Tip>

This example uses the [curl](https://curl.se/) utility on your local machine to call the Unstructured Partition Endpoint. It sends a source (input) file from your local machine to the Unstructured Partition Endpoint which then delivers the processed data to a destination (output) location, also on your local machine. Data is processed on Unstructured-hosted compute resources.

If you do not have a source file readily available, you could use for example a sample PDF file containing the text of the United States Constitution,
available for download from [https://constitutioncenter.org/media/files/constitution.pdf](https://constitutioncenter.org/media/files/constitution.pdf).

<Steps>
  <Step title="Set environment variables">
    From your terminal or Command Prompt, set the following two environment variables.

    * Replace `<your-unstructured-api-url>` with the Unstructured Partition Endpoint URL. This URL was provided to you when your Unstructured account was created.
      If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

      <Note>
        The default URL for the Unstructured Partition Endpoint is `https://api.unstructuredapp.io/general/v0/general`.
        However, you should always use the URL that was provided to you when your Unstructured account was created.
      </Note>

    * Replace `<your-unstructured-api-key>` with your Unstructured API key, which you generated earlier on this page.

    ```bash theme={null}
    export UNSTRUCTURED_API_URL=<your-unstructured-api-url>
    export UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
    ```
  </Step>

  <Step title="Run the curl command">
    Run the following `curl` command, replacing `<path/to/file>` with the path to the source file on your local machine.

    If the source file is not a PDF file, then remove `;type=application/pdf` from the final `--form` option in this command.

    ```bash theme={null}
    curl --request 'POST' \
    "$UNSTRUCTURED_API_URL" \
    --header 'accept: application/json' \
    --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
    --header 'content-Type: multipart/form-data' \
    --form 'content_type=string' \
    --form 'strategy=vlm' \
    --form 'vlm_model_provider=openai' \
    --form 'vlm_model=gpt-4o' \
    --form 'output_format=application/json' \
    --form 'files=@<path/to/file>;type=application/pdf'
    ```
  </Step>

  <Step title="Examine the results">
    After you run the `curl` command, the results are printed to your terminal or Command Prompt. The command might take several
    minutes to complete.

    By default, the JSON is printed without indenting or other whitespace. You can pretty-print the JSON output by using utilities such as [jq](https://jqlang.org/tutorial/) in future command runs.

    You can also pipe the JSON output to a local file by using the `curl` option [-o, --output \<file>](https://curl.se/docs/manpage.html#-o) in future command runs.
  </Step>
</Steps>

You can also call the Unstructured Partition Endpoint by using the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python) or the [Unstructured JavaScript/TypeScript SDK](/api-reference/legacy-api/partition/sdk-jsts).


# Partitioning strategies
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/partitioning



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

For certain document types, such as images and PDFs, for example, Unstructured products offer a variety of different
ways to preprocess them, controlled by the `strategy` parameter.

PDF documents, for example, vary in quality and complexity. In simple cases, traditional NLP extraction techniques may
be enough to extract all the text out of a document. In other cases, advanced image-to-text models are required
to process a PDF. You can think of the strategies as being "rule-based" workflows (thus they are "fast"), or
"model-based" workflows (slower workflow because it requires model inference, but you get "higher resolution", thus "hi\_res").
When choosing a partitioning strategy for your files, you have to be mindful of the quality/speed trade-off.
To give you an example, the `fast` strategy is roughly 100x faster than leading image-to-text models.

**Available options:**

* `auto` (default strategy): The "auto" strategy will choose the partitioning strategy based on document characteristics and the function kwargs.
* `fast`:  The "rule-based" strategy leverages traditional NLP extraction techniques to quickly pull all the text elements. "Fast" strategy is not recommended for image-based file types.
* `hi_res`: The "model-based" strategy identifies the layout of the document. The advantage of "hi\_res" is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements.
* `ocr_only`: Another "model-based" strategy that leverages Optical Character Recognition to extract text from the image-based files.
* `vlm`: Uses a vision language model (VLM) to extract text from these file types: `.bmp`, `.gif`, `.heic`, `.jpeg`, `.jpg`, `.pdf`, `.png`, `.tiff`, and `.webp`.


# Process an individual file by making a direct POST request
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/post-requests



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

Watch the following 4-minute video to learn how to make POST requests to the Unstructured Partition Endpoint to process individual files:

<iframe title="YouTube video player" />

Open the related [notebook](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_Partition_endpoint.ipynb) that is shown in the preceding video.

To make POST requests to the Unstructured Partition Endpoint, you will need:

These environment variables:

* `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
* `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

To get your API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

The API URL was provided to you when your Unstructured account was created.
If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  The default URL for the Unstructured Partition Endpoint is `https://api.unstructuredapp.io/general/v0/general`.
  However, you should always use the API URL that was provided to you when your Unstructured account was created.
</Note>

Let's start with a simple example in which you use [curl](https://curl.se/) to send a local PDF file (`*.pdf`) to partition via the Unstructured Partition Endpoint.

In this command, be sure to replace `<path/to/file>` with the path to your local PDF file.

```bash theme={null}
curl --request 'POST' \
"$UNSTRUCTURED_API_URL" \
--header 'accept: application/json' \
--header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
--header 'content-Type: multipart/form-data' \
--form 'content_type=string' \
--form 'strategy=vlm' \
--form 'vlm_model_provider=openai' \
--form 'vlm_model=gpt-4o' \
--form 'output_format=application/json' \
--form 'files=@<path/to/file>;type=application/pdf'
```

In the example above we're representing the API endpoint with the environment variable `UNSTRUCTURED_API_URL`. Note, however, that you also need to authenticate yourself with
your individual API Key, represented by the environment variable `UNSTRUCTURED_API_KEY`. Learn how to obtain an API URL and API key in the [Unstructured Partition Endpoint guide](/api-reference/legacy-api/partition/overview).

## Parameters & examples

The API parameters are the same across all methods of accessing the Unstructured Partition Endpoint.

* Refer to the [API parameters](/api-reference/legacy-api/partition/api-parameters) page for the full list of available parameters.
* Refer to the [Examples](/api-reference/legacy-api/partition/examples) page for some inspiration on using the parameters.

[//]: # "TODO: when we have the concepts page shared across products, link it from here for the users to learn about partition strategies, chunking strategies and other important shared concepts"

## Postman collection

Unstructured offers a [Postman collection](https://learning.postman.com/docs/collections/collections-overview/) that you can import into Postman to make POST requests through a graphical user interface.

1. [Install Postman](https://learning.postman.com/docs/getting-started/installation/installation-and-updates/).

2. [Sign in to Postman](https://learning.postman.com/docs/getting-started/installation/postman-account/#signing-in-to-postman).

3. In your workspace, click **Import**.

   <img alt="Import a Postman collection" />

4. In the **Paste cURL, Raw text or URL** box, enter the following URL, and then press `Enter`:

   ```
   https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/postman/Unstructured-POST.postman_collection.json
   ```

5. On the sidebar, click **Collections**.

6. Expand **Unstructured POST**.

7. Click **(Partition Endpoint) Basic Request**.

8. On the **Headers** tab, next to `unstructured-api-key`, enter your Unstructured API key in the **Value** column.

9. On the **Body** tab, next to `files`, click the **Select files** box in the **Value** column.

10. Click **New file from local machine**.

11. Browse to and select the file that you want Unstructured to process.

12. Click **Send**. Processing could take several minutes.

To download the processed data to your local machine, in the response area, click the ellipses, and then click **Save response to file**.


# Unstructured API Quickstart
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/quickstart



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

<Tip>
  Do you want to run this quickstart without modifying your local machine?
  [Skip ahead](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_Partition_Endpoint_Quickstart.ipynb) to run this quickstart as a notebook on Google Colab now!

  Do you want to just copy the sample code for use on your local machine? [Skip ahead](#sample-code) to the code now!

  This quickstart uses the Unstructured Partition Endpoint and focuses on a single, local file for ease-of-use demonstration purposes. This quickstart also
  focuses only on a limited set of Unstructured's full capabilities. To unlock the full feature set, as well as use Unstructured to do
  large-scale batch processing of multiple files and semi-structured data that are stored in remote locations,
  [skip over](/api-reference/workflow/overview#quickstart) to an expanded, advanced version of this quickstart that uses the
  Unstructured API's workflow operations instead.
</Tip>

<iframe title="YouTube video player" />

The following code shows how to use the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python)
to have Unstructured process one or more local files by using
the [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview).

To run this code, you will need the following:

* An Unstructured account and an Unstructured API key for your account. [Learn how](/api-reference/legacy-api/partition/overview#get-started).

* Python 3.9 or higher installed on your local machine.

* A Python virtual environment is recommended for isolating and versioning Python project code dependencies, but this is not required.
  To create and activate a virtual environment, you can use a framework such as
  [uv](https://docs.astral.sh/uv/) (recommended). Python provides a built-in framework named
  [venv](https://docs.python.org/3/library/venv.html).

* You must install the Unstructured Python SDK on your local machine, for example by running one of the
  following commands:

  * For `uv`, run `uv add unstructured-client`
  * For `venv` (or for no virtual environment), run `pip install unstructured-client`

* Add the following code to a Python file on your local machine; make the following code changes; and then run the code file to see the results.

  * Replace `<unstructured-api-key>` with your Unstructured API key.

  * To process all files within a directory, change `None` for `input_dir` to a string that contains the path to the directory on your local machine. This can be a relative or absolute path.

  * To process specific files within a directory or across multiple directories, change `None` for `input_file` to a string that contains
    a comma-separated list of filepaths on your local machine, for example `"./input/2507.13305v1.pdf,./input2/table-multi-row-column-cells.pdf"`. These filepaths
    can be relative or absolute.

    <Note>
      If `input_dir` and `input_file` are both set to something other than `None`, then the `input_dir` setting takes precedence, and the `input_file` setting is ignored.
    </Note>

  * For the `output_dir` parameter, specify a string that contains the path to the directory on your local machine that you want Unstructured to send its JSON output files. If the specified directory does not exist at that location, the code will create the missing directory for you. This path can be relative or absolute.

  <Note>
    If you choose to run this code in a notebook (such as a Google Colab notebook), you must do the following first to avoid
    nested event loop errors:

    1. Install the `nest_asyncio` Python package, by running the following command from a notebook cell:

       ```python theme={null}
       !pip install nest_asyncio
       ```

    2. Import the `nest_asyncio` package and then enable nested event loops, by running the following code from a notebook cell:

       ```python theme={null}
       import nest_asyncio

       nest_asyncio.apply()
       ```

    After completing these two steps, you can then run the following code as normal.
  </Note>

## Sample code

```python Python SDK theme={null}
import asyncio
import os
import json
import unstructured_client
from unstructured_client.models import shared, errors

client = unstructured_client.UnstructuredClient(
    api_key_auth="<unstructured-api-key>"
)

async def partition_file_via_api(filename):
    req = {
        "partition_parameters": {
            "files": {
                "content": open(filename, "rb"),
                "file_name": os.path.basename(filename),
            },
            "strategy": shared.Strategy.AUTO,
            "vlm_model": "gpt-4o",
            "vlm_model_provider": "openai",
            "languages": ['eng'],
            "split_pdf_page": True, 
            "split_pdf_allow_failed": True,
            "split_pdf_concurrency_level": 15
        }
    }

    try:
        res = await client.general.partition_async(request=req)
        return res.elements
    except errors.UnstructuredClientError as e:
        print(f"Error partitioning {filename}: {e.message}")
        return []

async def process_file_and_save_result(input_filename, output_dir):
    elements = await partition_file_via_api(input_filename)

    if elements:
        results_name = f"{os.path.basename(input_filename)}.json"
        output_filename = os.path.join(output_dir, results_name)

        with open(output_filename, "w") as f:
            json.dump(elements, f)

def load_filenames_in_directory(input_dir):
    filenames = []
    for root, _, files in os.walk(input_dir):
        for file in files:
            if not file.endswith('.json'):
                filenames.append(os.path.join(root, file))

    return filenames

async def process_files():
    # Initialize with either a directory name, to process everything in the dir,
    # or a comma-separated list of filepaths.
    input_dir = None   # "path/to/input/directory"
    input_files = None # "path/to/file,path/to/file,path/to/file"

    # Set to the directory for output json files. This dir 
    # will be created if needed.
    output_dir = "./output/"

    if input_dir:
        filenames = load_filenames_in_directory(input_dir)
    else:
        filenames = input_files.split(",")

    os.makedirs(output_dir, exist_ok=True)

    tasks = []
    for filename in filenames:
        tasks.append(
            process_file_and_save_result(filename, output_dir)
        )

    await asyncio.gather(*tasks)

if __name__ == "__main__":
    asyncio.run(process_files())
```

## Next steps

This quickstart shows how to use the Unstructured Partition Endpoint, which is intended for rapid prototyping of
some of Unstructured's [partitioning](/api-reference/legacy-api/partition/partitioning) strategies, with limited support for [chunking](/api-reference/legacy-api/partition/chunking).
It is designed to work only with processing of local files.

Take your code to the next level by switching over to the [Unstructured API's workflow operations](/api-reference/workflow/overview)
for production-level scenarios, file processing in batches, files and data in remote locations, full support for [chunking](/concepts/chunking),
generating [embeddings](/concepts/embedding), applying post-transform [enrichments](/concepts/enriching/overview),
using the latest and highest-performing models, and much more.
[Get started](/api-reference/workflow/overview).


# JavaScript/TypeScript SDK
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/sdk-jsts



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

The [Unstructured JavaScript/TypeScript SDK](https://github.com/Unstructured-IO/unstructured-js-client) client allows you to send one file at a time for processing by the Unstructured Partition Endpoint.

To use the JavaScript/TypeScript SDK, you'll first need to set an environment variable named `UNSTRUCTURED_API_KEY`,
representing your Unstructured API key. To get your API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

## Installation

Before using the SDK to interact with Unstructured, install the library:

<CodeGroup>
  ```bash JavaScript/TypeScript theme={null}
  npm install unstructured-client
  ```
</CodeGroup>

<Note>
  The SDK uses semantic versioning and major bumps could bring breaking changes. It is advised to
  pin your installed version.
</Note>

## Basics

Let's start with a simple example in which you send a PDF document to the Unstructured Partition Endpoint to be partitioned by Unstructured.

<Warning>
  The JavaScript/TypeScript SDK has the following breaking changes in v0.11.0:

  * Imports under the `dist` path have moved up a level
  * Enums are now used for parameters with a set of options
    * This includes `chunkingStrategy`, `outputFormat`, and `strategy`
  * All parameters to `partition` have moved to a `partitionParameters` object
</Warning>

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { UnstructuredClient } from "unstructured-client";
  import { PartitionResponse } from "unstructured-client/sdk/models/operations";
  import { Strategy } from "unstructured-client/sdk/models/shared";
  import * as fs from "fs";

  const key = process.env.UNSTRUCTURED_API_KEY;
  const url = process.env.UNSTRUCTURED_API_URL;

  const client = new UnstructuredClient({
      serverURL: url,
      security: {
          apiKeyAuth: key,
      },
  });

  const filename = "PATH_TO_INPUT_FILE";
  const data = fs.readFileSync(filename);

  client.general.partition({
      partitionParameters: {
          files: {
              content: data,
              fileName: filename,
          },
          strategy: Strategy.HiRes,
          splitPdfPage: true,
          splitPdfAllowFailed: true,
          splitPdfConcurrencyLevel: 15
          languages: ['eng']
      }
  }).then((res: PartitionResponse) => {
      if (res.statusCode == 200) {
          // Print the processed data's first element only.
          console.log(res.elements?.[0])

          // Write the processed data to a local file.
          const jsonElements = JSON.stringify(res, null, 2)

          fs.writeFileSync("PATH_TO_OUTPUT_FILE", jsonElements)
      }
  }).catch((e) => {
      if (e.statusCode) {
        console.log(e.statusCode);
        console.log(e.body);
      } else {
        console.log(e);
      }
  });
  ```

  ```typescript TypeScript (SDK <=v0.10.6) theme={null}
  import { UnstructuredClient } from "unstructured-client";
  import { PartitionResponse } from "unstructured-client/dist/sdk/models/operations";
  import * as fs from "fs";

  const key = process.env.UNSTRUCTURED_API_KEY;
  const url = process.env.UNSTRUCTURED_API_URL;

  const client = new UnstructuredClient({
      serverURL: url,
      security: {
          apiKeyAuth: key,
      },
  });

  const filename = "PATH_TO_FILE";
  const data = fs.readFileSync(filename);

  client.general.partition({
      files: {
          content: data,
          fileName: filename,
      },
      strategy: 'hi_res',
      languages: ['eng']
  }).then((res: PartitionResponse) => {
      if (res.statusCode == 200) {
          console.log(res.elements?.[0]);
      }
  }).catch((e) => {
    if (e.statusCode) {
      console.log(e.statusCode);
      console.log(e.body);
    } else {
      console.log(e);
    }
  });
  ```
</CodeGroup>

For a code example that works with an entire directory of files instead of just a single PDF, see the [Processing multiple files](#processing-multiple-files) section.

## Page splitting

In order to speed up processing of large PDF files, the `splitPdfPage`[\*](#parameter-names) parameter is `true` by default. This
causes the PDF to be split into small batches of pages before sending requests to the API. The client
awaits all parallel requests and combines the responses into a single response object. This is specific to PDF files and other
filetypes are ignored.

The number of parallel requests is controlled by `splitPdfConcurrencyLevel`[\*](#parameter-names).
The default is 8 and the max is set to 15 to avoid high resource usage and costs.

If at least one request is successful, the responses are combined into a single response object. An
error is returned only if all requests failed or there was an error during splitting.

<Note>
  This feature may lead to unexpected results when chunking because the server does not see the entire
  document context at once. If you'd like to chunk across the whole document and still get the speedup from
  parallel processing, you can:

  * Partition the PDF with `splitPdfPage` set to `true`, without any chunking parameters.
  * Store the returned elements in `results.json`.
  * Partition this JSON file with the desired chunking parameters.
</Note>

```typescript TypeScript theme={null}
client.general.partition({
    partitionParameters: {
        files: {
            content: data,
            fileName: filename,
        },
        strategy: Strategy.HiRes,
        // Set to `false` to disable PDF splitting
        splitPdfPage: true,
        // Continue PDF splitting even if some earlier split operations fail.
        splitPdfAllowFailed: true,
        // Modify splitPdfConcurrencyLevel to set the number of parallel requests
        splitPdfConcurrencyLevel: 10,
    }
})
```

## Customizing the client

### Retries

You can also change the defaults for retries through the `retryConfig`[\*](#parameter-names)
when initializing the client. If a request to the API fails, the client will retry the
request with an exponential backoff strategy up to a maximum interval of one minute. The
function keeps retrying until the total elapsed time exceeds `maxElapsedTime`[\*](#parameter-names),
which defaults to one hour:

```typescript TypeScript theme={null}
const key = process.env.UNSTRUCTURED_API_KEY;
const url = process.env.UNSTRUCTURED_API_URL;    

const client = new UnstructuredClient({
    security: { apiKeyAuth: key },
    serverURL: url,
    retryConfig: {
        strategy: "backoff",
        retryConnectionErrors: true,
        backoff: {
            initialInterval: 500,
            maxInterval: 60000,
            exponent: 1.5,
            maxElapsedTime: 900000, // 15min*60sec*1000ms = 15 minutes
        },
    };
});
```

## Processing multiple files

The code example in the [Basics](#basics) section processes a single PDF file. But what if you want to process
multiple files inside a directory with a mixture of subdirectories and files with different file types?

The following example takes an input directory path to read files from and an output directory path to
write the processed data to, processing one file at a time.

```typescript TypeScript theme={null}
import { UnstructuredClient } from "unstructured-client";
import * as fs from "fs";
import * as path from "path";
import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
import { PartitionResponse } from "unstructured-client/sdk/models/operations";

// Send all files in the source path to Unstructured for processing.
// Send the processed data to the destination path.
function processFiles(
    client: UnstructuredClient,
    sourcePath: string,
    destinationPath: string
): void {

    // If an output directory does not exist for the corresponding input
    // directory, then create it.
    if (!fs.existsSync(destinationPath)) {
        fs.mkdirSync(destinationPath, { recursive: true });
    }

    // Get all folders and files at the current level of the input directory.
    const items = fs.readdirSync(sourcePath);

    // For each folder and file in the input directory...
    for (const item of items) {
        const inputPath = path.join(sourcePath, item);
        const outputPath = path.join(destinationPath, item)

        // If it's a folder, call this function recursively.
        if (fs.statSync(inputPath).isDirectory()) {
            processFiles(client, inputPath, outputPath);
        } else {
            // If it's a file, send it to Unstructured for processing.
            const data = fs.readFileSync(inputPath);

            client.general.partition({
                partitionParameters: {
                    files: {
                        content: data,
                        fileName: inputPath
                    },
                    strategy: Strategy.HiRes,
                    splitPdfPage: true,
                    splitPdfConcurrencyLevel: 15,
                    splitPdfAllowFailed: true
                }
            }).then((res: PartitionResponse) => {
                // If successfully processed, write the processed data to
                // the destination directory.
                if (res.statusCode == 200) {
                    const jsonElements = JSON.stringify(res, null, 2)
                    fs.writeFileSync(outputPath + ".json", jsonElements)
                }
            }).catch((e) => {
                if (e.statusCode) {
                    console.log(e.statusCode);
                    console.log(e.body);
                } else {
                    console.log(e);
                }
            });
        }
    }
}

const client = new UnstructuredClient({
    security: { apiKeyAuth: process.env.UNSTRUCTURED_API_KEY },
    serverURL: process.env.UNSTRUCTURED_API_URL
});

processFiles(
    client,
    process.env.LOCAL_FILE_INPUT_DIR,
    process.env.LOCAL_FILE_OUTPUT_DIR
);
```

## Parameters & examples

The parameter names used in this document are for the JavaScript/TypeScript SDK, which follows the `camelCase`
convention. The Python SDK follows the `snake_case` convention. Other than this difference in naming convention,
the names used in the SDKs are the same across all methods.

* Refer to the [API parameters](/api-reference/legacy-api/partition/api-parameters) page for the full list of available parameters.
* Refer to the [Examples](/api-reference/legacy-api/partition/examples) page for some inspiration on using the parameters.


# Python SDK
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/sdk-python



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

The [Unstructured Python SDK](https://github.com/Unstructured-IO/unstructured-python-client) client allows you to send one file at a time for processing by
the [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview).

To use the Python SDK, you'll first need to set an environment variable named `UNSTRUCTURED_API_KEY`,
representing your Unstructured API key. To get your API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

## Installation

Before using the SDK to interact with Unstructured, install the library:

```bash Python theme={null}
pip install unstructured-client
```

<Note>
  The SDK uses semantic versioning and major bumps could bring breaking changes. It is advised to
  pin your installed version. See the [migration guide](#migration-guide), later on this page, for breaking change announcements.
</Note>

## Basics

Let's start with a simple example in which you send a PDF document to the Unstructured Partition Endpoint to be partitioned by Unstructured.

```python Python theme={null}
import os, json

import unstructured_client
from unstructured_client.models import operations, shared

client = unstructured_client.UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
)

filename = "PATH_TO_INPUT_FILE"

req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=shared.Files(
            content=open(filename, "rb"),
            file_name=filename,
        ),
        strategy=shared.Strategy.VLM,
        vlm_model="gpt-4o",
        vlm_model_provider="openai",
        languages=['eng'],
        split_pdf_page=True,            # If True, splits the PDF file into smaller chunks of pages.
        split_pdf_allow_failed=True,    # If True, the partitioning continues even if some pages fail.
        split_pdf_concurrency_level=15  # Set the number of concurrent request to the maximum value: 15.
    ),
)

try:
    res = client.general.partition(
        request=req
    )
    element_dicts = [element for element in res.elements]
    
    # Print the processed data's first element only.
    print(element_dicts[0])

    # Write the processed data to a local file.
    json_elements = json.dumps(element_dicts, indent=2)

    with open("PATH_TO_OUTPUT_FILE", "w") as file:
        file.write(json_elements)
except Exception as e:
    print(e)
```

## Async partitioning

The Python SDK also has a `partition_async`. This call is equivalent to `partition` except that it can be used in a non blocking context. For instance, `asyncio.gather` can be used to concurrently process multiple files inside of a directory hierarchy, as demonstrated here:

<CodeGroup>
  ```python Python theme={null}
  import asyncio
  import os
  import json
  import unstructured_client
  from unstructured_client.models import shared

  client = unstructured_client.UnstructuredClient(
      api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
  )

  async def call_api(filename, input_dir, output_dir):
      req = {
          "partition_parameters": {
              "files": {
                  "content": open(filename, "rb"),
                  "file_name": os.path.basename(filename),
              },
              "strategy": shared.Strategy.VLM,
              "vlm_model": "gpt-4o",
              "vlm_model_provider": "openai",
              "languages": ['eng'],
              "split_pdf_page", True,            # If True, splits the PDF file into smaller chunks of pages.
              "split_pdf_allow_failed": True,    # If True, the partitioning continues even if some pages fail.
              "split_pdf_concurrency_level": 15  # Set the number of concurrent request to the maximum value: 15.
          }
      }

      try:
          res = await client.general.partition_async(
              request=req
          )
          element_dicts = [element for element in res.elements]
          json_elements = json.dumps(element_dicts, indent=2)

          # Create the output directory structure.
          relative_path = os.path.relpath(os.path.dirname(filename), input_dir)
          output_subdir = os.path.join(output_dir, relative_path)
          os.makedirs(output_subdir, exist_ok=True)

          # Write the output file.
          output_filename = os.path.join(output_subdir, os.path.basename(filename) + ".json")
          with open(output_filename, "w") as file:
              file.write(json_elements)

      except Exception as e:
          print(f"Error processing {filename}: {e}")

  async def process_files(input_directory, output_directory):
      tasks = []

      for root, _, files in os.walk(input_directory):
          for file in files:
              if not file.endswith('.json'):
                  full_path = os.path.join(root, file)
                  tasks.append(call_api(full_path, input_directory, output_directory))

      await asyncio.gather(*tasks)

  if __name__ == "__main__":
      asyncio.run(process_files(
          input_directory=os.getenv("LOCAL_FILE_INPUT_DIR"), 
          output_directory=os.getenv("LOCAL_FILE_OUTPUT_DIR")
      ))
  ```
</CodeGroup>

## Page splitting

In order to speed up processing of large PDF files, the `split_pdf_page`[\*](#parameter-names) parameter is `True` by default. This
causes the PDF to be split into small batches of pages before sending requests to the API. The client
awaits all parallel requests and combines the responses into a single response object. This is specific to PDF files and other
filetypes are ignored.

The number of parallel requests is controlled by `split_pdf_concurrency_level`[\*](#parameter-names).
The default is 8 and the max is set to 15 to avoid high resource usage and costs.

If at least one request is successful, the responses are combined into a single response object. An
error is returned only if all requests failed or there was an error during splitting.

<Note>
  This feature may lead to unexpected results when chunking because the server does not see the entire
  document context at once. If you'd like to chunk across the whole document and still get the speedup from
  parallel processing, you can:

  * Partition the PDF with `split_pdf_page` set to `True`, without any chunking parameters.
  * Store the returned elements in `results.json`.
  * Partition this JSON file with the desired chunking parameters.
</Note>

```python Python theme={null}
req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=shared.Files(
            content=file.read(),
            file_name=filename,
        ),
        strategy=shared.Strategy.VLM,
        vlm_model="gpt-4o",
        vlm_model_provider="openai",
        split_pdf_page=True,            # If True, splits the PDF file into smaller chunks of pages.
        split_pdf_allow_failed=True,    # If True, the partitioning continues even if some pages fail.
        split_pdf_concurrency_level=15  # Set the number of concurrent request to the maximum value: 15.
    )
)
res = client.general.partition(
    request=req
)
```

## Customizing the client

### Retries

You can also change the defaults for retries through the `retry_config`[\*](#parameter-names)
parameter when initializing the client. If a request to the API fails, the client will retry the
request with an exponential backoff strategy up to a maximum interval of one minute. The
function keeps retrying until the total elapsed time exceeds `max_elapsed_time`[\*](#parameter-names),
which defaults to one hour. However, you can override these defaults, for example as follows:

```python Python theme={null}
# ...
from unstructured_client.utils import RetryConfig, BackoffStrategy

client = UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    retry_config=RetryConfig(
        strategy="backoff",
        retry_connection_errors=True,
        backoff=BackoffStrategy(
            initial_interval=500,     # 0.5 seconds: 500 milliseconds
            max_interval=60000,       # 1 minute: 60 seconds * 1000 milliseconds
            exponent=1.5,
            max_elapsed_time=900000,  # 15 minutes: 15 * 60 seconds * 1000 milliseconds
        ),
    )
)
```

Available `RetryConfig` settings include:

| Setting                            | Description                                                                       |
| ---------------------------------- | --------------------------------------------------------------------------------- |
| `strategy`                         | The strategy to use for retries. The only supported value is `backoff`.           |
| `retry_connection_errors`          | True to retry on connection errors.                                               |
| `BackoffStrategy.initial_interval` | After the first error, wait the specified number of milliseconds before retrying. |
| `BackoffStrategy.max_interval`     | The maximum wait time, specified in milliseconds between retries.                 |
| `BackoffStrategy.exponent`         | After each retry, increase the wait time exponentially by the specified factor.   |
| `BackoffStrategy.max_elapsed_time` | Stop retrying after the specified number of milliseconds.                         |

### Disabling SSL validation

If you disable SSL validation, requests will accept any TLS certificate
presented by the server and ignore hostname mismatches and/or expired certificates,
which will make your application vulnerable to man-in-the-middle (MitM) attacks.
Only set this to `False` for testing.

```python Python theme={null}
http_client = requests.Session()
http_client.verify = False

client = UnstructuredClient(
    client=http_client,
    ...
)
```

## Handling the response

The partition response defaults to a dict format that can be converted to Unstructured elements with
the `elements_from_dicts` utility function as seen below. Otherwise, the API response can be sent directly
to your vector store or another destination.

```python Python theme={null}
from unstructured.staging.base import elements_from_dicts

# ...

if res.elements is not None:
    elements = elements_from_dicts(response.elements)
```

## Parameters & examples

The parameter names used in this document are for the Python SDK, which follow snake\_case convention. The JavaScript/TypeScript SDK follows camelCase
convention. Other than this difference in naming convention,
the names used in the SDKs are the same across all methods.

* Refer to the [API parameters](/api-reference/legacy-api/partition/api-parameters) page for the full list of available parameters.
* Refer to the [Examples](/api-reference/legacy-api/partition/examples) page for some inspiration on using the parameters.

## Migration guide

There are breaking changes beginning with Python SDK version 0.26.0. If you encounter any errors when upgrading, please find the solution below.

**If you see the error: `AttributeError: 'PartitionParameters' object has no attribute 'partition_parameters'`**

Before 0.26.0, the SDK accepted a `PartitionParameters` object as input to the `sdk.general.partition` function. Beginning with 0.26.0, this object must be wrapped in a `PartitionRequest` object. The old behavior was deprecated in 0.23.0 and removed in 0.26.0.

```python theme={null}
# Instead of:
from unstructured_client.models import shared

req = shared.PartitionParameters(
    files=files,
)

resp = s.general.partition(
    request=req
)

# Switch to:
from unstructured_client.models import shared, operations

req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=files,
    )
)

resp = s.general.partition(
    request=req
)
```

**If you see the error: `TypeError: BaseModel.__init__() takes 1 positional argument but 2 were given`**

Beginning with 0.26.0, the `PartitionRequest` constructor no longer allows for positional arguments. You must specify `partition_parameters` by name.

```python theme={null}
# Instead of:
req = operations.PartitionRequest(
    shared.PartitionParameters(
        files=files,
    )
)

# Switch to:
req = operations.PartitionRequest(
    partition_parameters=shared.PartitionParameters(
        files=files,
    )
)
```

**If you see the error: `TypeError: General.partition() takes 1 positional argument but 2 were given`**

Beginning with 0.26.0, the `partition` function no longer allows for positional arguments. You must specify `request` by name.

```python theme={null}
# Instead of:
resp = s.general.partition(req)

# Switch to:
resp = s.general.partition(
    request=req
)
```


# Speed up processing of large files and batches
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/speed-up-large-files-batches



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

When you use Unstructured, here are some techniques that you can try to help speed up the processing of large files and large batches of files.

Choose your partitioning strategy wisely. For example, if you have simple PDFs that don't have images and tables, you might be able to use the `fast` strategy. Try the `fast` strategy on a few of your documents before you try using the `hi_res` strategy. [Learn more](/api-reference/legacy-api/partition/partitioning).

To speed up PDF file processing, the [Unstructured SDK for Python](/api-reference/legacy-api/partition/sdk-python) and the [Unstructured SDK for JavaScript/TypeScript](/api-reference/legacy-api/partition/sdk-jsts) provide the following parameters to help speed up processing a large PDF file:

* `split_pdf_page` (Python) or `splitPdfPage` (JavaScript/TypeScript), when set to true (the default), splits the PDF file on the client side before sending it as batches to Unstructured for processing. The number of pages in each batch is determined internally. Batches can contain between 2 and 20 pages.
* `split_pdf_concurrency_level` (Python) or `splitPdfConcurrencyLevel` (JavaScript/TypeScript) is an integer that specifies the number of parallel requests. The default is 5. The maximum is 15. This behavior is ignored unless `split_pdf_page` (Python) or `splitPdfPage` (JavaScript/TypeScript) is also set to true.
* `split_pdf_allow_failed` (Python) or splitPdfAllowFailed\` (JavaScript/TypeScript), when set to true, allows partitioning to continue even if some pages fail.
* `split_pdf_page_range` (Python only) is a list of two integers that specify the beginning and ending page numbers of the PDF file to be sent. A `ValueError` is raised if the specified range is not valid. This behavior is ignored unless `split_pdf_page` is also set to true.

[Learn more](/api-reference/legacy-api/partition/sdk-python#page-splitting).


# Extract tables as HTML
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/text-as-html



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

## Task

You want to get, save, or show the contents of elements that are represented as HTML, such as tables that are embedded in a PDF document.

## Approach

Extract the contents of an element's `text_as_html` JSON object, which is nested inside of its parent `metadata` object.

## To run this example

You will need a document that is one of the document types that can output the `text_as_html` JSON object. For the list of applicable document types, see the entries in the table at the beginning of [Partitioning](/open-source/core-functionality/partitioning) where "Table Support" is "Yes."

This example uses a PDF file with an embedded table.

## Code

For the [Unstructured Python SDK](/api-reference/legacy-api/partition/sdk-python), you'll need:

These environment variables:

* `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
* `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

```python Python SDK theme={null}
from unstructured_client import UnstructuredClient
from unstructured_client.models import operations, shared
from unstructured.staging.base import elements_from_dicts, elements_to_json

import os, webbrowser

if __name__ == "__main__":
    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    # Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/pdf/embedded-images-tables.pdf

    # Where to get the local file, relative to this .py file.
    local_input_filepath = "local-ingest-input-pdf/embedded-images-tables.pdf"

    # Where to store the retrieved HTML (and the processed JSON), relative to this .py file.
    local_output_filepath = "local-ingest-output"

    with open(local_input_filepath, "rb") as f:
        files = shared.Files(
            content=f.read(),
            file_name=local_input_filepath
        )

    request = operations.PartitionRequest(
        shared.PartitionParameters(
            files=files,
            strategy=shared.Strategy.VLM,
            vlm_model="gpt-4o",
            vlm_model_provider="openai",
            split_pdf_page=True,
            split_pdf_allow_failed=True,
            split_pdf_concurrency_level=15
        )
    )

    try:
        result = await client.general.partition_async(
            request=request
        )

        # Provide some minimal CSS for better table readability.
        table_css = "<head><style>table, th, td { border: 1px solid; }</style></head>"

        for element in result.elements:
            if "text_as_html" in element["metadata"]:
                # Surround the element's HTML with basic <html> and <body> tags, and add the minimal CSS.
                html_string = f"<!DOCTYPE html><html>{table_css}<body>{element["metadata"]["text_as_html"]}</body></html>"

                # Save the element's HTML to a local file.
                save_path = f"{local_output_path}/{element["element_id"]}.html"
                file = open(save_path, 'w')
                file.write(html_string)
                file.close()

                # View the locally saved file in the local default web browser.
                webbrowser.open_new(f"file:///{os.getcwd()}/{save_path}")

        # Also get the elements for inspection and validation.
        dict_elements = elements_from_dicts(
            element_dicts=result.elements
        )

        # Save the elements as JSON.
        elements_to_json(
            elements=dict_elements,
            indent=2,
            filename=f"{local_output_filepath}/embedded-images-tables.json"
        )
    except Exception as e:
        print(e)
```

## See also

* [Extract images and tables from documents](/api-reference/legacy-api/partition/extract-image-block-types)
* [Table Extraction from PDF](/examplecode/codesamples/apioss/table-extraction-from-pdf)


# Transform a JSON file into a different schema
Source: https://docs.unstructured.io/api-reference/legacy-api/partition/transform-schemas



<Note>
  The following information applies to the legacy Unstructured Partition Endpoint.

  Unstructured recommends that you use the
  [on-demand jobs](/api-reference/api/job/create-job) functionality in the
  [Unstructured API](/api-reference/overview) instead. Unstructured's on-demand jobs provide
  many benefits over the legacy Unstructured Partition Endpoint, including support for:

  * Production-level usage.
  * Multiple local input files in batches.
  * The latest and highest-performing models.
  * Post-transform enrichments.
  * All of Unstructured's chunking strategies.
  * The generation of vector embeddings.

  The Unstructured API also provides support for processing files and data in remote locations.
</Note>

## Task

You want to convert a JSON file that Unstructured produces into a separate JSON file that uses
a different JSON schema than the one that Unstructured uses.

## Approach

Use a Python package such as [json-converter](https://pypi.org/project/json-converter/) in your Python code project
to transform your source JSON file into a target JSON file that conforms to your own schema.

<Info>The `json-converter` package is not owned or supported by Unstructured. For questions and
requests, see the [Issues](https://github.com/ebi-ait/json-converter/issues) tab of the
`json-converter` repository in GitHub.</Info>

## Code

<Steps>
  <Step title="Install dependencies">
    In your local Python code project, install the [json-converter](https://pypi.org/project/json-converter/)
    package.

    ```bash theme={null}
    pip install json-converter
    ```
  </Step>

  <Step title="Identify the JSON file to transform">
    1. Find the local source JSON file that you want to transform.
    2. Note the JSON field names and structures that you want to transform. For example, the JSON file might
       look like the following (the ellipses indicate content omitted for brevity):

       ```json theme={null}
       [
         {
           "type": "...",
           "element_id": "...",
           "text": "...",
           "metadata": {
             "filetype": "...",
             "languages": [
               "eng"
             ],
             "page_number": 1,
             "filename": "..."
           }
         },
         {
           "type": "...",
           "element_id": "...",
           "text": "...",
           "metadata": {
             "filetype": "...",
             "languages": [
               "eng"
             ],
             "page_number": 1,
             "filename": "..."
           }
         },
         {
           "...": "..."
         }
       ]
       ```
  </Step>

  <Step title="Create the JSON field mappings file">
    1. Decide what you want the JSON schema in the transformed file to look like. For example, the
       transformed JSON file might look like the following (the ellipses indicate content omitted for brevity):

       ```json theme={null}
       [
         {
           "content_type": "...",
           "content_id": "...",
           "content": "...",
           "content_properties": {
             "page": 1
           }
         },
         {
           "content_type": "...",
           "content_id": "...",
           "content": "...",
           "content_properties": {
             "page": 1
           }
         },
         {
           "...": "..."
         }
       ]
       ```

    2. Create the JSON field mappings file, for example:

       ```json theme={null}
       {
         "content_type": ["type"],
         "content_id": ["element_id"],
         "content": ["text"]
         "content_properties.page": ["metadata.page_number"]
       }
       ```

       This file declares the following mappings:

       * The `type` field is renamed to `content_type`.
       * The `element_id` field is renamed to `content_id`.
       * The `text` field is renamed to `content`.
       * The `page_number` field nested inside `metadata` is renamed to `page` and is nested inside `content_properties`.
       * All of the other fields (`filetype`, `languages`, and `filename`) are dropped.

       For more information about the format of this JSON field mappings file, see the
       [Project Description](https://pypi.org/project/json-converter) in the `json-converter` page on PyPI or the
       [README](https://github.com/ebi-ait/json-converter) in the `json-converter` repository in GitHub.
  </Step>

  <Step title="Add and run the transform code">
    1. Set the following local environment variables:

       * Set `LOCAL_FILE_INPUT_PATH` to the local path to the source JSON file.
       * Set `LOCAL_FILE_OUTPUT_PATH` to the local path to the target JSON file.
       * Set `LOCAL_FIELD_MAPPINGS_PATH` to the local path to the JSON field mappings file.

    2. Add the following Python code file to your project:

       ```python theme={null}
       import os, json
       from json_converter.json_mapper import JsonMapper

       # Converts one JSON file with one schema to another 
       # JSON file with a different schema. 
       # Provide the path to the input (source) and output 
       # (target) JSON files and the path to the JSON schema 
       # field mappings file.
       def convert_json_with_schemas(
           input_file_path: str,
           output_file_path: str,
           field_mappings_path: str
       ) -> None:

           output_data = []

           try:
               with open(field_mappings_path, 'r') as f:
                   element_mappings = json.load(f)
           except FileNotFoundError:
               print(f"Error: Input JSON schema field mappings file '{input_file_path}' not found.")
           except IOError as e:
               print(f"I/O error occurred: {e}")
           except Exception as e:
               print(f"An unexpected error occurred: {e}")

           try:
               with open(input_file_path, 'r') as f:
                   input_data = json.load(f)
                   
                   for item in input_data:
                       converted_data = JsonMapper(item).map(element_mappings)
                       output_data.append(converted_data)
           except FileNotFoundError:
               print(f"Error: Input JSON file '{input_file_path}' not found.")
           except IOError as e:
               print(f"I/O error occurred: {e}")
           except Exception as e:
               print(f"An unexpected error occurred: {e}")

           try:
               with open(output_file_path, 'w') as f:
                   json.dump(output_data, f, indent=2)
           
               print(f"Transformation complete. Output written to '{output_file_path}'.")
           except IOError as e:
               print(f"I/O error occurred: {e}")
           except Exception as e:
               print(f"An unexpected error occurred: {e}")

       if __name__ == "__main__":
           convert_json_with_schemas(
               input_file_path=os.getenv("LOCAL_FILE_INPUT_PATH"),
               output_file_path=os.getenv("LOCAL_FIELD_MAPPINGS_PATH"),
               field_mappings_path=os.getenv("LOCAL_FILE_OUTPUT_PATH")
           )
       ```

    3. Run the Python code file.

    4. Check the path specified by `LOCAL_FILE_OUTPUT_PATH` for the transformed JSON file.
  </Step>
</Steps>

## Troubleshooting

### Error when trying to import Mapping from collections

**Issue**: When you run your Python code file, the following error message appears: "ImportError: cannot import name 'Mapping' from 'collections'".

**Cause**: When you use the `json-converter` package with newer versions of Python such as 3.11 and later,
Python tries to use an outdated import in this `json-converter` package.

**Solution**: Update the `json-converter` package's source code to use a different import, as follows:

1. In your Python project, find the `json-converter` package's source location, by running the `pip show` command:

   ```bash theme={null}
   pip show json-converter
   ```

   Note the path in the **Location** field.

2. Use your code editor to the open the path to the `json-converter` package's source code.

3. In the source code, open the file named `json_mapper.py`.

4. Change the following line of code...

   ```python theme={null}
   from collections import Mapping
   ```

   ...to the following line of code, by adding `.abc`:

   ```python theme={null}
   from collections.abc import Mapping
   ```

5. Save this source code file.

6. Run your Python code file again.


# Overview
Source: https://docs.unstructured.io/api-reference/overview



<Tip>
  To start using the Unstructured API right away, skip ahead to the [quickstart](/api-reference/quickstart/overview) now!
</Tip>

The Unstructured API provides a set of programmatic operations that enable you to have Unstructured:

* Perform a full range of partitioning, enrichment, chunking, and embedding of your files and data, producing results that are AI-ready.
* Batch-process files locally; and batch-process files and data that are stored in remote storage locations, databases, and vector stores.
* Retrieve batch-processed local file results locally as well; and send batch-processed remote file or data results to remote storage locations, databases, and vector stores.
* Use the latest and highest-performing models.

<Frame>
  <img alt="Unstructured API example" />
</Frame>

# Benefits over open source

The Unstructured API provides the following benefits beyond the [Unstructured open source library](/open-source/introduction/overview) offering:

* Designed for production scenarios.
* Significantly increased performance on document and table extraction.
* Access to newer and more sophisticated vision transformer models.
* Access to Unstructured's fine-tuned OCR models.
* Access to Unstructured's by-page and by-similarity chunking strategies.
* Adherence to security and SOC 2 Type 1, SOC 2 Type 2, HIPAA, GDPR, ISO 27001, FedRAMP, and CMMC 2.0 Level 2 compliance standards. For details, see the [Unstructured Trust Portal](https://trust.unstructured.io/).
* Authentication and identity management.
* Incremental data loading.
* Image extraction from documents.
* More sophisticated document hierarchy detection.
* Unstructured manages code dependencies, for instance for libraries such as Tesseract.
* Unstructured manages its own infrastructure, including parallelization and other performance optimizations.

[Learn more](/open-source/introduction/overview#limits).

## Pricing

To call the Unstructured API, you must have an Unstructured account.

Unstructured offers different account types with different pricing plans:

* <Icon icon="person" />  **Let's Go** and **Pay-As-You-Go** - A single user, with a single workspace, hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="building" />  **Business** - Multiple users and workspaces, with three options:

  * <Icon icon="people-group" />  **Business SaaS** - Hosted alongside other accounts on Unstructured's cloud infrastructure.
  * <Icon icon="shield-halved" />  **Dedicated instance** - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control.
  * <Icon icon="shield" />  **In-VPC** - Hosted within your own VPC on your own cloud infrastructure.

  **Business** accounts also allow for robust customization of Unstructured's features for your unique needs.

For more details, see the [Unstructured Pricing](https://unstructured.io/pricing) page.

To upgrade your account from **Let's Go** or **Pay-As-You-Go** to **Business**,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Some of these plans have billing details that are determined on a per-page basis.

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

## Get support

For technical support for Unstructured accounts, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

## Learn more

* <Icon icon="blog" />  [Unstructured API: All Unstructured Enterprise ETL Functionality for Your MCP Integrations](https://unstructured.io/blog/unstructured-api-all-unstructured-enterprise-etl-functionality-for-your-mcp-integrations)


# Embed quickstart
Source: https://docs.unstructured.io/api-reference/quickstart/embed

Create an on-demand job to partition, chunk, and embed local files into AI-ready JSON with vector embeddings for semantic search.

[Complete the requirements](/api-reference/quickstart/overview#requirements) before you begin.

You can also learn about Unstructured's [partitioning](/concepts/partitioning), [chunking](/concepts/chunking), and [embedding](/concepts/embedding) before you begin.

<Steps>
  <Step title="Create the on-demand job">
    Replace `INPUT_DIR` with the path to your local directory of files to process. The response includes the job ID.

    <Note>
      Each on-demand job is limited to 10 files, and each file is limited to 50 MB in size.

      If you need to launch a series of on-demand jobs in rapid succession, you must wait at least one second between launch
      requests. Otherwise, you will receive a rate limit error.

      A maximum of 5 on-demand jobs can be running in your Unstructured account. If you launch a new on-demand job
      but 5 existing on-demand jobs are still running, the new on-demand job will remain in a scheduled state until one of the 5
      existing on-demand jobs is done running.
    </Note>

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        INPUT_DIR="/full/path/to/your/directory"

        form_args=()
        for filepath in "$INPUT_DIR"/*; do
            [ -f "$filepath" ] || continue
            filename=$(basename "$filepath")
            mimetype=$(file --mime-type -b "$filepath")
            form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
        done

        response=$(curl --request POST --location \
          "$UNSTRUCTURED_API_URL/jobs/" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --form 'request_data={"job_nodes":[{"name":"Partitioner","type":"partition","subtype":"vlm","settings":{"is_dynamic":true,"allow_fast":true}},{"name":"Chunker","type":"chunk","subtype":"chunk_by_title","settings":{"max_characters":2048,"new_after_n_chars":1500,"overlap":160}},{"name":"Embedder","type":"embed","subtype":"azure_openai","settings":{"model_name":"text-embedding-3-large"}}]}' \
          "${form_args[@]}")

        JOB_ID=$(echo "$response" | jq -r '.id')
        echo "Job ID: $JOB_ID"
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import mimetypes
        import os
        import json

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import CreateJobRequest
        from unstructured_client.models.shared import BodyCreateJob, InputFiles

        INPUT_DIR = "/full/path/to/your/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        input_files = []
        for filename in os.listdir(INPUT_DIR):
            full_path = os.path.join(INPUT_DIR, filename)
            if not os.path.isfile(full_path):
                continue
            content_type, _ = mimetypes.guess_type(full_path)
            input_files.append(
                InputFiles(
                    content=open(full_path, "rb"),
                    file_name=filename,
                    content_type=content_type or "application/octet-stream"
                )
            )

        response = client.jobs.create_job(
            request=CreateJobRequest(
                body_create_job=BodyCreateJob(
                    request_data=json.dumps({
                        "job_nodes": [
                            {
                                "name": "Partitioner",
                                "type": "partition",
                                "subtype": "vlm",
                                "settings": {
                                    "is_dynamic": True,
                                    "allow_fast": True
                                }
                            },
                            {
                                "name": "Chunker",
                                "type": "chunk",
                                "subtype": "chunk_by_title",
                                "settings": {
                                    "max_characters": 2048,
                                    "new_after_n_chars": 1500,
                                    "overlap": 160
                                }
                            },
                            {
                                "name": "Embedder",
                                "type": "embed",
                                "subtype": "azure_openai",
                                "settings": {
                                    "model_name": "text-embedding-3-large"
                                }
                            }
                        ]
                    }),
                    input_files=input_files
                )
            )
        )

        job_info = response.job_information
        print(f"Job ID: {job_info.id}")
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Poll for job status">
    Replace `JOB_ID` with the job ID from the previous step. This script polls every 10 seconds and stops when the job completes.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"

        while true; do
            job=$(curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
              --header "accept: application/json" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

            status=$(echo "$job" | jq -r '.status')
            echo "Job status: $status"

            if [ "$status" = "COMPLETED" ]; then
                echo "Job completed."
                echo "Output node file IDs: $(echo "$job" | jq -c '[.output_node_files[].file_id]')"
                break
            elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
                echo "Job did not complete successfully: $status"
                exit 1
            fi

            sleep 10
        done
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import os
        import time

        from unstructured_client import UnstructuredClient

        JOB_ID = "<job-id>"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        while True:
            response = client.jobs.get_job(request={"job_id": JOB_ID})
            job_info = response.job_information
            status = job_info.status

            print(f"Job status: {status.value}")

            if status == "COMPLETED":
                print("Job completed.")
                print(f"Output node file IDs: {[f.file_id for f in (job_info.output_node_files or [])]}")
                break
            elif status in ("FAILED", "STOPPED"):
                raise RuntimeError(f"Job did not complete successfully: {status}")

            time.sleep(10)
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Download the job output">
    Replace `JOB_ID`, `OUTPUT_FILE_IDS`, and `OUTPUT_DIR` with your values from the previous steps.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"
        OUTPUT_FILE_IDS=("<output-file-id>" "<output-file-id>") # From Step 1
        OUTPUT_DIR="/full/path/to/your/output/directory"

        mkdir -p "$OUTPUT_DIR"

        for file_id in "${OUTPUT_FILE_IDS[@]}"; do
            curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
              --output "$OUTPUT_DIR/$file_id.json"
            echo "Saved: $OUTPUT_DIR/$file_id.json"
        done
        ```
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import json
        import os

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import DownloadJobOutputRequest

        JOB_ID = "<job-id>"
        OUTPUT_FILE_IDS = ["<output-file-id>", "<output-file-id>"]] # From Step 1
        OUTPUT_DIR = "/full/path/to/your/output/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        os.makedirs(OUTPUT_DIR, exist_ok=True)

        for file_id in OUTPUT_FILE_IDS:
            response = client.jobs.download_job_output(
                request=DownloadJobOutputRequest(job_id=JOB_ID, file_id=file_id)
            )
            output_path = os.path.join(OUTPUT_DIR, f"{file_id}.json")
            with open(output_path, "w") as f:
                json.dump(response.any, f, indent=4)
            print(f"Saved: {output_path}")
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Complete end-to-end script

Replace `INPUT_DIR` and `OUTPUT_DIR` with your directory paths, then save and run this script.

<Tabs>
  <Tab title="curl">
    This script requires [jq](https://jqlang.org/download/) to parse JSON responses.

    ```bash theme={null}
    #!/usr/bin/env bash

    INPUT_DIR="/full/path/to/your/input/directory"
    OUTPUT_DIR="/full/path/to/your/output/directory"

    # Step 1: Create the on-demand job.
    form_args=()
    for filepath in "$INPUT_DIR"/*; do
        [ -f "$filepath" ] || continue
        filename=$(basename "$filepath")
        mimetype=$(file --mime-type -b "$filepath")
        form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
    done

    response=$(curl --request POST --location \
      "$UNSTRUCTURED_API_URL/jobs/" \
      --header "accept: application/json" \
      --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
      --form 'request_data={"job_nodes":[{"name":"Partitioner","type":"partition","subtype":"vlm","settings":{"is_dynamic":true,"allow_fast":true}},{"name":"Chunker","type":"chunk","subtype":"chunk_by_title","settings":{"max_characters":2048,"new_after_n_chars":1500,"overlap":160}},{"name":"Embedder","type":"embed","subtype":"azure_openai","settings":{"model_name":"text-embedding-3-large"}}]}' \
      "${form_args[@]}")

    JOB_ID=$(echo "$response" | jq -r '.id')
    echo "Job ID: $JOB_ID"

    # Step 2: Poll until the job completes.
    output_file_ids=()

    while true; do
        job=$(curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

        status=$(echo "$job" | jq -r '.status')
        echo "Job status: $status"

        if [ "$status" = "COMPLETED" ]; then
            echo "Job completed."

            while IFS= read -r id; do
                output_file_ids+=("$id")
            done < <(echo "$job" | jq -r '.output_node_files[].file_id')

            break
        elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
            echo "Job did not complete successfully: $status"
            exit 1
        fi

        sleep 10
    done

    # Step 3: Download the job output.
    mkdir -p "$OUTPUT_DIR"

    for file_id in "${output_file_ids[@]}"; do
        curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --output "$OUTPUT_DIR/$file_id.json"
        echo "Saved: $OUTPUT_DIR/$file_id.json"
    done
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import json
    import mimetypes
    import os
    import time

    from unstructured_client import UnstructuredClient
    from unstructured_client.models.operations import CreateJobRequest, DownloadJobOutputRequest
    from unstructured_client.models.shared import BodyCreateJob, InputFiles

    INPUT_DIR = "/full/path/to/your/input/directory"
    OUTPUT_DIR = "/full/path/to/your/output/directory"

    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL")
    )

    # Step 1: Create the on-demand job.
    input_files = []
    for filename in os.listdir(INPUT_DIR):
        full_path = os.path.join(INPUT_DIR, filename)
        if not os.path.isfile(full_path):
            continue
        content_type, _ = mimetypes.guess_type(full_path)
        input_files.append(
            InputFiles(
                content=open(full_path, "rb"),
                file_name=filename,
                content_type=content_type or "application/octet-stream"
            )
        )

    response = client.jobs.create_job(
        request=CreateJobRequest(
            body_create_job=BodyCreateJob(
                request_data=json.dumps({
                    "job_nodes": [
                        {
                            "name": "Partitioner",
                            "type": "partition",
                            "subtype": "vlm",
                            "settings": {
                                "is_dynamic": True,
                                "allow_fast": True
                            }
                        },
                        {
                            "name": "Chunker",
                            "type": "chunk",
                            "subtype": "chunk_by_title",
                            "settings": {
                                "max_characters": 2048,
                                "new_after_n_chars": 1500,
                                "overlap": 160
                            }
                        },
                        {
                            "name": "Embedder",
                            "type": "embed",
                            "subtype": "azure_openai",
                            "settings": {
                                "model_name": "text-embedding-3-large"
                            }
                        }
                    ]
                }),
                input_files=input_files
            )
        )
    )

    job_id = response.job_information.id
    print(f"Job ID: {job_id}")

    # Step 2: Poll until the job completes.
    while True:
        response = client.jobs.get_job(request={"job_id": job_id})
        job_info = response.job_information
        status = job_info.status

        print(f"Job status: {status.value}")

        if status == "COMPLETED":
            print("Job completed.")
            break
        elif status in ("FAILED", "STOPPED"):
            raise RuntimeError(f"Job did not complete successfully: {status}")

        time.sleep(10)

    output_node_file_ids = [f.file_id for f in (job_info.output_node_files or [])]

    # Step 3: Download the job output.
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    for file_id in output_node_file_ids:
        response = client.jobs.download_job_output(
            request=DownloadJobOutputRequest(job_id=job_id, file_id=file_id)
        )
        output_path = os.path.join(OUTPUT_DIR, f"{file_id}.json")
        with open(output_path, "w") as f:
            json.dump(response.any, f, indent=4)
        print(f"Saved: {output_path}")
    ```
  </Tab>
</Tabs>

## What's next?

* Create on-demand jobs that [only partition](/api-reference/quickstart/transform), add [enrichments](/api-reference/quickstart/enrich), or perform [structured data extractions](/api-reference/quickstart/extract).
* Learn about the Unstructured API's other [workflow operations](/api-reference/workflow/overview).


# Enrich quickstart
Source: https://docs.unstructured.io/api-reference/quickstart/enrich

Create an on-demand job to partition and enrich local files into AI-ready JSON with image descriptions, table HTML, named entities, and generative OCR output.

[Complete the requirements](/api-reference/quickstart/overview#requirements) before you begin.

You can also learn about Unstructured's [partitioning](/concepts/partitioning) and [enriching](/concepts/enriching/overview) before you begin.

<Steps>
  <Step title="Create the on-demand job">
    Replace `INPUT_DIR` with the path to your local directory of files to process. The response includes the job ID.

    <Note>
      Each on-demand job is limited to 10 files, and each file is limited to 50 MB in size.

      If you need to launch a series of on-demand jobs in rapid succession, you must wait at least one second between launch
      requests. Otherwise, you will receive a rate limit error.

      A maximum of 5 on-demand jobs can be running in your Unstructured account. If you launch a new on-demand job
      but 5 existing on-demand jobs are still running, the new on-demand job will remain in a scheduled state until one of the 5
      existing on-demand jobs is done running.
    </Note>

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        INPUT_DIR="/full/path/to/your/directory"

        form_args=()
        for filepath in "$INPUT_DIR"/*; do
            [ -f "$filepath" ] || continue
            filename=$(basename "$filepath")
            mimetype=$(file --mime-type -b "$filepath")
            form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
        done

        response=$(curl --request POST --location \
          "$UNSTRUCTURED_API_URL/jobs/" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --form 'request_data={"job_nodes":[{"name":"Partitioner","type":"partition","subtype":"unstructured_api","settings":{"strategy":"hi_res","extract_image_block_types":["image","table"]}},{"name":"Generative OCR","type":"prompter","subtype":"openai_ocr","settings":{"provider_type":"openai","model":"gpt-5-mini"}},{"name":"Image Description","type":"prompter","subtype":"openai_image_description","settings":{"provider_type":"openai","model":"gpt-5-mini"}},{"name":"Table to HTML","type":"prompter","subtype":"twopass_table2html"},{"name":"Table Description","type":"prompter","subtype":"openai_table_description","settings":{"provider_type":"openai","model":"gpt-5-mini"}},{"name":"NER","type":"prompter","subtype":"openai_ner","settings":{"provider_type":"openai","model":"gpt-5-mini"}}]}' \
          "${form_args[@]}")

        JOB_ID=$(echo "$response" | jq -r '.id')
        echo "Job ID: $JOB_ID"
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import mimetypes
        import os
        import json

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import CreateJobRequest
        from unstructured_client.models.shared import BodyCreateJob, InputFiles

        INPUT_DIR = "/full/path/to/your/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        input_files = []
        for filename in os.listdir(INPUT_DIR):
            full_path = os.path.join(INPUT_DIR, filename)
            if not os.path.isfile(full_path):
                continue
            content_type, _ = mimetypes.guess_type(full_path)
            input_files.append(
                InputFiles(
                    content=open(full_path, "rb"),
                    file_name=filename,
                    content_type=content_type or "application/octet-stream"
                )
            )

        response = client.jobs.create_job(
            request=CreateJobRequest(
                body_create_job=BodyCreateJob(
                    request_data=json.dumps({
                        "job_nodes": [
                            {
                                "name": "Partitioner",
                                "type": "partition",
                                "subtype": "unstructured_api",
                                "settings": {
                                    "strategy": "hi_res",
                                    "extract_image_block_types": ["image", "table"]
                                }
                            },
                            {
                                "name": "Generative OCR",
                                "type": "prompter",
                                "subtype": "openai_ocr",
                                "settings": {
                                    "provider_type": "openai",
                                    "model": "gpt-5-mini"
                                }
                            },
                            {
                                "name": "Image Description",
                                "type": "prompter",
                                "subtype": "openai_image_description",
                                "settings": {
                                    "provider_type": "openai",
                                    "model": "gpt-5-mini"
                                }
                            },
                            {
                                "name": "Table to HTML",
                                "type": "prompter",
                                "subtype": "twopass_table2html"
                            },
                            {
                                "name": "Table Description",
                                "type": "prompter",
                                "subtype": "openai_table_description",
                                "settings": {
                                    "provider_type": "openai",
                                    "model": "gpt-5-mini"
                                }
                            },
                            {
                                "name": "NER",
                                "type": "prompter",
                                "subtype": "openai_ner",
                                "settings": {
                                    "provider_type": "openai",
                                    "model": "gpt-5-mini"
                                }
                            }
                        ]
                    }),
                    input_files=input_files
                )
            )
        )

        job_info = response.job_information
        print(f"Job ID: {job_info.id}")
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Poll for job status">
    Replace `JOB_ID` with the job ID from the previous step. This script polls every 10 seconds and stops when the job completes.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"

        while true; do
            job=$(curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
              --header "accept: application/json" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

            status=$(echo "$job" | jq -r '.status')
            echo "Job status: $status"

            if [ "$status" = "COMPLETED" ]; then
                echo "Job completed."
                echo "Output node file IDs: $(echo "$job" | jq -c '[.output_node_files[].file_id]')"
                break
            elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
                echo "Job did not complete successfully: $status"
                exit 1
            fi

            sleep 10
        done
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import os
        import time

        from unstructured_client import UnstructuredClient

        JOB_ID = "<job-id>"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        while True:
            response = client.jobs.get_job(request={"job_id": JOB_ID})
            job_info = response.job_information
            status = job_info.status

            print(f"Job status: {status.value}")

            if status == "COMPLETED":
                print("Job completed.")
                print(f"Output node file IDs: {[f.file_id for f in (job_info.output_node_files or [])]}")
                break
            elif status in ("FAILED", "STOPPED"):
                raise RuntimeError(f"Job did not complete successfully: {status}")

            time.sleep(10)
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Download the job output">
    Replace `JOB_ID`, `OUTPUT_FILE_IDS`, and `OUTPUT_DIR` with your values from the previous steps.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"
        OUTPUT_FILE_IDS=("<output-file-id>" "<output-file-id>") # From Step 1
        OUTPUT_DIR="/full/path/to/your/output/directory"

        mkdir -p "$OUTPUT_DIR"

        for file_id in "${OUTPUT_FILE_IDS[@]}"; do
            curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
              --output "$OUTPUT_DIR/$file_id.json"
            echo "Saved: $OUTPUT_DIR/$file_id.json"
        done
        ```
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import json
        import os

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import DownloadJobOutputRequest

        JOB_ID = "<job-id>"
        OUTPUT_FILE_IDS = ["<output-file-id>", "<output-file-id>"] # From Step 1
        OUTPUT_DIR = "/full/path/to/your/output/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        os.makedirs(OUTPUT_DIR, exist_ok=True)

        for file_id in OUTPUT_FILE_IDS:
            response = client.jobs.download_job_output(
                request=DownloadJobOutputRequest(job_id=JOB_ID, file_id=file_id)
            )
            output_path = os.path.join(OUTPUT_DIR, file_id, ".json")
            with open(output_path, "w") as f:
                json.dump(response.any, f, indent=4)
            print(f"Saved: {output_path}")
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Complete end-to-end script

Replace `INPUT_DIR` and `OUTPUT_DIR` with your directory paths, then save and run this script.

<Tabs>
  <Tab title="curl">
    This script requires [jq](https://jqlang.org/download/) to parse JSON responses.

    ```bash theme={null}
    #!/usr/bin/env bash

    INPUT_DIR="/full/path/to/your/input/directory"
    OUTPUT_DIR="/full/path/to/your/output/directory"

    # Step 1: Create the on-demand job.
    form_args=()
    for filepath in "$INPUT_DIR"/*; do
        [ -f "$filepath" ] || continue
        filename=$(basename "$filepath")
        mimetype=$(file --mime-type -b "$filepath")
        form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
    done

    response=$(curl --request POST --location \
      "$UNSTRUCTURED_API_URL/jobs/" \
      --header "accept: application/json" \
      --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
      --form 'request_data={"job_nodes":[{"name":"Partitioner","type":"partition","subtype":"unstructured_api","settings":{"strategy":"hi_res","extract_image_block_types":["image","table"]}},{"name":"Generative OCR","type":"prompter","subtype":"openai_ocr","settings":{"provider_type":"openai","model":"gpt-5-mini"}},{"name":"Image Description","type":"prompter","subtype":"openai_image_description","settings":{"provider_type":"openai","model":"gpt-5-mini"}},{"name":"Table to HTML","type":"prompter","subtype":"twopass_table2html"},{"name":"Table Description","type":"prompter","subtype":"openai_table_description","settings":{"provider_type":"openai","model":"gpt-5-mini"}},{"name":"NER","type":"prompter","subtype":"openai_ner","settings":{"provider_type":"openai","model":"gpt-5-mini"}}]}' \
      "${form_args[@]}")

    JOB_ID=$(echo "$response" | jq -r '.id')
    echo "Job ID: $JOB_ID"

    # Step 2: Poll until the job completes.
    output_file_ids=()

    while true; do
        job=$(curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

        status=$(echo "$job" | jq -r '.status')
        echo "Job status: $status"

        if [ "$status" = "COMPLETED" ]; then
            echo "Job completed."

            while IFS= read -r id; do
                output_file_ids+=("$id")
            done < <(echo "$job" | jq -r '.output_node_files[].file_id')

            break
        elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
            echo "Job did not complete successfully: $status"
            exit 1
        fi

        sleep 10
    done

    # Step 3: Download the job output.
    mkdir -p "$OUTPUT_DIR"

    for file_id in "${output_file_ids[@]}"; do
        curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --output "$OUTPUT_DIR/$file_id.json"
        echo "Saved: $OUTPUT_DIR/$file_id.json"
    done
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import json
    import mimetypes
    import os
    import time

    from unstructured_client import UnstructuredClient
    from unstructured_client.models.operations import CreateJobRequest, DownloadJobOutputRequest
    from unstructured_client.models.shared import BodyCreateJob, InputFiles

    INPUT_DIR = "/full/path/to/your/input/directory"
    OUTPUT_DIR = "/full/path/to/your/output/directory"

    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL")
    )

    # Step 1: Create the on-demand job.
    input_files = []
    for filename in os.listdir(INPUT_DIR):
        full_path = os.path.join(INPUT_DIR, filename)
        if not os.path.isfile(full_path):
            continue
        content_type, _ = mimetypes.guess_type(full_path)
        input_files.append(
            InputFiles(
                content=open(full_path, "rb"),
                file_name=filename,
                content_type=content_type or "application/octet-stream"
            )
        )

    response = client.jobs.create_job(
        request=CreateJobRequest(
            body_create_job=BodyCreateJob(
                request_data=json.dumps({
                    "job_nodes": [
                        {
                            "name": "Partitioner",
                            "type": "partition",
                            "subtype": "unstructured_api",
                            "settings": {
                                "strategy": "hi_res",
                                "extract_image_block_types": ["image", "table"]
                            }
                        },
                        {
                            "name": "Generative OCR",
                            "type": "prompter",
                            "subtype": "openai_ocr",
                            "settings": {
                                "provider_type": "openai",
                                "model": "gpt-5-mini"
                            }
                        },
                        {
                            "name": "Image Description",
                            "type": "prompter",
                            "subtype": "openai_image_description",
                            "settings": {
                                "provider_type": "openai",
                                "model": "gpt-5-mini"
                            }
                        },
                        {
                            "name": "Table to HTML",
                            "type": "prompter",
                            "subtype": "twopass_table2html"
                        },
                        {
                            "name": "Table Description",
                            "type": "prompter",
                            "subtype": "openai_table_description",
                            "settings": {
                                "provider_type": "openai",
                                "model": "gpt-5-mini"
                            }
                        },
                        {
                            "name": "NER",
                            "type": "prompter",
                            "subtype": "openai_ner",
                            "settings": {
                                "provider_type": "openai",
                                "model": "gpt-5-mini"
                            }
                        }
                    ]
                }),
                input_files=input_files
            )
        )
    )

    job_id = response.job_information.id
    print(f"Job ID: {job_id}")

    # Step 2: Poll until the job completes.
    while True:
        response = client.jobs.get_job(request={"job_id": job_id})
        job_info = response.job_information
        status = job_info.status

        print(f"Job status: {status.value}")

        if status == "COMPLETED":
            print("Job completed.")
            break
        elif status in ("FAILED", "STOPPED"):
            raise RuntimeError(f"Job did not complete successfully: {status}")

        time.sleep(10)

    output_node_file_ids = [f.file_id for f in (job_info.output_node_files or [])]

    # Step 3: Download the job output.
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    for file_id in output_node_file_ids:
        response = client.jobs.download_job_output(
            request=DownloadJobOutputRequest(job_id=job_id, file_id=file_id)
        )
        output_path = os.path.join(OUTPUT_DIR, f"{file_id}.json")
        with open(output_path, "w") as f:
            json.dump(response.any, f, indent=4)
        print(f"Saved: {output_path}")
    ```
  </Tab>
</Tabs>

## What's next?

* Create on-demand jobs that [only partition](/api-reference/quickstart/transform), add [embeddings](/api-reference/quickstart/embed), or perform [structured data extractions](/api-reference/quickstart/extract).
* Learn about the Unstructured API's other [workflow operations](/api-reference/workflow/overview).


# Extract quickstart
Source: https://docs.unstructured.io/api-reference/quickstart/extract

Create an on-demand job to partition local files and extract specified data as JSON object data records using the Unstructured API.

[Complete the requirements](/api-reference/quickstart/overview#requirements) before you begin.

You can also learn about Unstructured's [partitioning](/concepts/partitioning) and [structured data extraction](/concepts/structured-data-extractor/data-extractor) before you begin.

<Steps>
  <Step title="Create the on-demand job">
    Replace `EXTRACTION_PROMPT` with your extraction prompt, and `INPUT_DIR` with the path to your local directory of files to process. The response includes the job ID.

    <Note>
      Each on-demand job is limited to 10 files, and each file is limited to 50 MB in size.

      If you need to launch a series of on-demand jobs in rapid succession, you must wait at least one second between launch
      requests. Otherwise, you will receive a rate limit error.

      A maximum of 5 on-demand jobs can be running in your Unstructured account. If you launch a new on-demand job
      but 5 existing on-demand jobs are still running, the new on-demand job will remain in a scheduled state until one of the 5
      existing on-demand jobs is done running.
    </Note>

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        EXTRACTION_PROMPT="Represent dates such as May-12-24 as 2024-05-12 and June-12-25 as 2025-06-12."
        INPUT_DIR="/full/path/to/your/directory"

        form_args=()

        for filepath in "$INPUT_DIR"/*; do
            [ -f "$filepath" ] || continue
            filename=$(basename "$filepath")
            mimetype=$(file --mime-type -b "$filepath")
            form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
        done

        json_schema='{"type":"object","properties":{"invoice_number":{"type":"number"},"invoice_date":{"type":"string"},"payment_due":{"type":"string"},"bill_to":{"type":"string"}},"additionalProperties":false,"required":["invoice_number","invoice_date","payment_due","bill_to"]}'

        request_data=$(jq -n --arg prompt "$EXTRACTION_PROMPT" --arg schema "$json_schema" '{
        "job_nodes": [
            {"name":"Partitioner","type":"partition","subtype":"vlm","settings":{"is_dynamic":true,"allow_fast":true}},
            {"name":"Extractor","type":"structured_data_extractor","subtype":"llm","settings":{"schema_to_extract":{"json_schema":$schema,"extraction_guidance":$prompt},"provider":"openai","model":"gpt-5-mini","output_mode":"extracted_data_only"}}
        ]
        }')

        response=$(curl --request POST --location \
          "$UNSTRUCTURED_API_URL/jobs/" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --form "request_data=$request_data" \
          "${form_args[@]}")

        JOB_ID=$(echo "$response" | jq -r '.id')
        echo "Job ID: $JOB_ID"
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import mimetypes
        import os
        import json

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import CreateJobRequest
        from unstructured_client.models.shared import BodyCreateJob, InputFiles

        EXTRACTION_PROMPT = "<your-extraction-prompt>"
        INPUT_DIR = "/full/path/to/your/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        input_files = []
        for filename in os.listdir(INPUT_DIR):
            full_path = os.path.join(INPUT_DIR, filename)
            if not os.path.isfile(full_path):
                continue
            content_type, _ = mimetypes.guess_type(full_path)
            input_files.append(
                InputFiles(
                    content=open(full_path, "rb"),
                    file_name=filename,
                    content_type=content_type or "application/octet-stream"
                )
            )

        response = client.jobs.create_job(
            request=CreateJobRequest(
                body_create_job=BodyCreateJob(
                    request_data=json.dumps({
                        "job_nodes": [
                            {
                                "name": "Partitioner",
                                "type": "partition",
                                "subtype": "vlm",
                                "settings": {
                                    "is_dynamic": True,
                                    "allow_fast": True
                                }
                            },
                            {
                                "name": "Extractor",
                                "type": "structured_data_extractor",
                                "subtype": "llm",
                                "settings": {
                                    "schema_to_extract": {
                                        "json_schema": {
                                            "type": "object",
                                            "properties": {
                                                "invoice_number": { "type": "number" },
                                                "invoice_date": { "type": "string" },
                                                "payment_due": { "type": "string" },
                                                "bill_to": { "type": "string" }
                                            },
                                            "additionalProperties": False,
                                            "required": [
                                                "invoice_number",
                                                "invoice_date",
                                                "payment_due",
                                                "bill_to"
                                            ]
                                        },
                                        "extraction_guidance": EXTRACTION_PROMPT
                                    },
                                    "provider": "openai",
                                    "model": "gpt-5-mini",
                                    "output_mode": "extracted_data_only"
                                }
                            }
                        ]
                    }),
                    input_files=input_files
                )
            )
        )

        job_info = response.job_information
        print(f"Job ID: {job_info.id}")
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Poll for job status">
    Replace `JOB_ID` with the job ID from the previous step. This script polls every 10 seconds and stops when the job completes.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"

        while true; do
            job=$(curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
              --header "accept: application/json" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

            status=$(echo "$job" | jq -r '.status')
            echo "Job status: $status"

            if [ "$status" = "COMPLETED" ]; then
                echo "Job completed."
                echo "Output node file IDs: $(echo "$job" | jq -c '[.output_node_files[].file_id]')"
                break
            elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
                echo "Job did not complete successfully: $status"
                exit 1
            fi

            sleep 10
        done
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import os
        import time

        from unstructured_client import UnstructuredClient

        JOB_ID = "<job-id>"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        while True:
            response = client.jobs.get_job(request={"job_id": JOB_ID})
            job_info = response.job_information
            status = job_info.status

            print(f"Job status: {status.value}")

            if status == "COMPLETED":
                print("Job completed.")
                print(f"Output node file IDs: {[f.file_id for f in (job_info.output_node_files or [])]}")
                break
            elif status in ("FAILED", "STOPPED"):
                raise RuntimeError(f"Job did not complete successfully: {status}")

            time.sleep(10)
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Download the job output">
    Replace `JOB_ID`, `OUTPUT_FILE_IDS`, and `OUTPUT_DIR` with your values from the previous steps.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"
        OUTPUT_FILE_IDS=("<output-file-id>" "<output-file-id>") # From Step 1
        OUTPUT_DIR="/full/path/to/your/output/directory"

        mkdir -p "$OUTPUT_DIR"

        for file_id in "${OUTPUT_FILE_IDS[@]}"; do
            curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
              --output "$OUTPUT_DIR/$file_id.json"
            echo "Saved: $OUTPUT_DIR/$file_id.json"
        done
        ```
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import json
        import os

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import DownloadJobOutputRequest

        JOB_ID = "<job-id>"
        OUTPUT_FILE_IDS = ["<output-file-id>", "<output-file-id>"] # From Step 1
        OUTPUT_DIR = "/full/path/to/your/output/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        os.makedirs(OUTPUT_DIR, exist_ok=True)

        for file_id in OUTPUT_FILE_IDS:
            response = client.jobs.download_job_output(
                request=DownloadJobOutputRequest(job_id=JOB_ID, file_id=file_id)
            )
            output_path = os.path.join(OUTPUT_DIR, file_id, ".json")
            with open(output_path, "w") as f:
                json.dump(response.any, f, indent=4)
            print(f"Saved: {output_path}")
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Complete end-to-end script

Replace `EXTRACTION_PROMPT`, `INPUT_DIR`, and `OUTPUT_DIR` with your values, then save and run this script.

<Tabs>
  <Tab title="curl">
    This script requires [jq](https://jqlang.org/download/) to parse JSON responses.

    ```bash theme={null}
    #!/usr/bin/env bash

    EXTRACTION_PROMPT="Represent dates such as May-12-24 as 2024-05-12 and June-12-25 as 2025-06-12."
    INPUT_DIR="/full/path/to/your/input/directory"
    OUTPUT_DIR="/full/path/to/your/output/directory"

    # Step 1: Create the on-demand job.
    form_args=()

    for filepath in "$INPUT_DIR"/*; do
        [ -f "$filepath" ] || continue
        filename=$(basename "$filepath")
        mimetype=$(file --mime-type -b "$filepath")
        form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
    done

    json_schema='{"type":"object","properties":{"invoice_number":{"type":"number"},"invoice_date":{"type":"string"},"payment_due":{"type":"string"},"bill_to":{"type":"string"}},"additionalProperties":false,"required":["invoice_number","invoice_date","payment_due","bill_to"]}'

    request_data=$(jq -n --arg prompt "$EXTRACTION_PROMPT" --arg schema "$json_schema" '{
    "job_nodes": [
        {"name":"Partitioner","type":"partition","subtype":"vlm","settings":{"is_dynamic":true,"allow_fast":true}},
        {"name":"Extractor","type":"structured_data_extractor","subtype":"llm","settings":{"schema_to_extract":{"json_schema":$schema,"extraction_guidance":$prompt},"provider":"openai","model":"gpt-5-mini","output_mode":"extracted_data_only"}}
    ]
    }')

    response=$(curl --request POST --location \
      "$UNSTRUCTURED_API_URL/jobs/" \
      --header "accept: application/json" \
      --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
      --form "request_data=$request_data" \
      "${form_args[@]}")

    JOB_ID=$(echo "$response" | jq -r '.id')
    echo "Job ID: $JOB_ID"

    # Step 2: Poll until the job completes.
    output_file_ids=()

    while true; do
        job=$(curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

        status=$(echo "$job" | jq -r '.status')
        echo "Job status: $status"

        if [ "$status" = "COMPLETED" ]; then
            echo "Job completed."

            while IFS= read -r id; do
                output_file_ids+=("$id")
            done < <(echo "$job" | jq -r '.output_node_files[].file_id')

            break
        elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
            echo "Job did not complete successfully: $status"
            exit 1
        fi

        sleep 10
    done

    # Step 3: Download the job output.
    mkdir -p "$OUTPUT_DIR"

    for file_id in "${output_file_ids[@]}"; do
        curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --output "$OUTPUT_DIR/$file_id.json"
        echo "Saved: $OUTPUT_DIR/$file_id.json"
    done
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import json
    import mimetypes
    import os
    import time

    from unstructured_client import UnstructuredClient
    from unstructured_client.models.operations import CreateJobRequest, DownloadJobOutputRequest
    from unstructured_client.models.shared import BodyCreateJob, InputFiles

    EXTRACTION_PROMPT = "<your-extraction-prompt>"
    INPUT_DIR = "/full/path/to/your/input/directory"
    OUTPUT_DIR = "/full/path/to/your/output/directory"

    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL")
    )

    # Step 1: Create the on-demand job.
    input_files = []
    for filename in os.listdir(INPUT_DIR):
        full_path = os.path.join(INPUT_DIR, filename)
        if not os.path.isfile(full_path):
            continue
        content_type, _ = mimetypes.guess_type(full_path)
        input_files.append(
            InputFiles(
                content=open(full_path, "rb"),
                file_name=filename,
                content_type=content_type or "application/octet-stream"
            )
        )

    response = client.jobs.create_job(
        request=CreateJobRequest(
            body_create_job=BodyCreateJob(
                request_data=json.dumps({
                    "job_nodes": [
                        {
                            "name": "Partitioner",
                            "type": "partition",
                            "subtype": "vlm",
                            "settings": {
                                "is_dynamic": True,
                                "allow_fast": True
                            }
                        },
                        {
                            "name": "Extractor",
                            "type": "structured_data_extractor",
                            "subtype": "llm",
                            "settings": {
                                "schema_to_extract": {
                                    "json_schema": {
                                        "type": "object",
                                        "properties": {
                                            "invoice_number": { "type": "number" },
                                            "invoice_date": { "type": "string" },
                                            "payment_due": { "type": "string" },
                                            "bill_to": { "type": "string" }
                                        },
                                        "additionalProperties": False,
                                        "required": [
                                            "invoice_number",
                                            "invoice_date",
                                            "payment_due",
                                            "bill_to"
                                        ]
                                    },
                                    "extraction_guidance": EXTRACTION_PROMPT
                                },
                                "provider": "openai",
                                "model": "gpt-5-mini",
                                "output_mode": "extracted_data_only"
                            }
                        }
                    ]
                }),
                input_files=input_files
            )
        )
    )

    job_id = response.job_information.id
    print(f"Job ID: {job_id}")

    # Step 2: Poll until the job completes.
    while True:
        response = client.jobs.get_job(request={"job_id": job_id})
        job_info = response.job_information
        status = job_info.status

        print(f"Job status: {status.value}")

        if status == "COMPLETED":
            print("Job completed.")
            break
        elif status in ("FAILED", "STOPPED"):
            raise RuntimeError(f"Job did not complete successfully: {status}")

        time.sleep(10)

    output_node_file_ids = [f.file_id for f in (job_info.output_node_files or [])]

    # Step 3: Download the job output.
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    for file_id in output_node_file_ids:
        response = client.jobs.download_job_output(
            request=DownloadJobOutputRequest(job_id=job_id, file_id=file_id)
        )
        output_path = os.path.join(OUTPUT_DIR, f"{file_id}.json")
        with open(output_path, "w") as f:
            json.dump(response.any, f, indent=4)
        print(f"Saved: {output_path}")
    ```
  </Tab>
</Tabs>

## What's next?

* Create on-demand jobs that [only partition](/api-reference/quickstart/transform), add [enrichments](/api-reference/quickstart/enrich), or add [embeddings](/api-reference/quickstart/embed).
* Learn about the Unstructured API's other [workflow operations](/api-reference/workflow/overview).


# Unstructured API quickstart
Source: https://docs.unstructured.io/api-reference/quickstart/overview

Use curl or the Unstructured Python SDK to turn local documents into AI-ready JSON.

## Requirements

<Steps>
  <Step title="Sign up">
    **For individual developers and individual users**: If you have not yet signed up for Unstructured, do so now as follows:

    1. Go to [https://developer.unstructured.io](https://developer.unstructured.io).
    2. Follow the on-screen instructions to sign up.

    **For Business account users**: If you are an [Unstructured Business](/business/overview) account user, you are already signed up. Skip ahead to the next step.
  </Step>

  <Step title="Get your API URL">
    **For individual developers and individual users**: Get your API URL as follows:

    1. Go to [https://platform.unstructured.io](https://platform.unstructured.io) and sign in.
    2. On the sidebar, click **API Keys**.
    3. Next to **Unstructured API Endpoint**, copy the complete URL.
    4. Skip ahead to the next step.

    **For Business account users**: Get your API URL as follows:

    1. Go to your Unstructured account's website and sign in. If you don't know this website's address, see your
       Unstructured account administrator.
    2. In the top navigation bar's organization account drop-down, select the name of the Unstructured organizational
       account that contains the workspace for which you want to get the API URL.
    3. In the sidebar, click **Settings**, and then click **View All Workspaces**.
    4. Click the name of the workspace for which you want to get the API URL.
    5. Next to **Unstructured API Endpoint**, copy the complete URL.
  </Step>

  <Step title="Get your API key">
    **For individual developers and individual users**: Get your API key as follows:

    1. Go to [https://developer.unstructured.io](https://developer.unstructured.io) and sign in.
    2. Click **API Keys**, and store the API key's value in a secure location.
    3. Skip ahead to the next step.

    **For Business account users**: Get your API key as follows:

    1. Go to your Unstructured account's website and sign in. If you don't know this website's address, see your
       Unstructured account administrator.
    2. In the top navigation bar's organization account drop-down, select the name of the Unstructured organizational
       account that contains the workspace for which you want to create an API key.
    3. In the sidebar, click **Settings**, and then click **View All Workspaces**.
    4. Click the name of the workspace for which you want to create an API key.
    5. In the workspace's title bar, click **API Keys**.
    6. Click **Generate New Key**. This API key works for this workspace only.
    7. Enter some name for new API key, and then click **Continue**.
    8. Next to **Key ID** for the new API key, click the **Copy** icon.
    9. Paste the copied API key's value into a secure location.
  </Step>

  <Step title="Set environment variables">
    This quickstart assumes that you have set two local environment variables that your local curl scripts or Python code can
    access. These two environment variables are:

    * `UNSTRUCTURED_API_URL`, representing your Unstructured API URL, which you got from the previous step.
    * `UNSTRUCTURED_API_KEY`, representing your Unstructured API key, which you got from the previous step.

    For instructions about how to set these environment variables accordingly, see your local operating system's documentation.
  </Step>

  <Step title="Install the tools">
    **For curl users**: To use this quickstart with curl, you only need to have curl installed locally. For installation instructions, see your local operating system's documentation.

    **For Python users**: To use this quickstart with Python, install the Unstructured Python SDK locally by running the following command:

    ```bash theme={null}
    pip install unstructured-client
    ```
  </Step>
</Steps>

## Quickstarts by use case

<CardGroup>
  <Card title="Transform" href="/api-reference/quickstart/transform">
    Transform your documents' contents into text and metadata, which Unstructured outputs as standard AI-ready JSON.
  </Card>

  <Card title="Enrich" href="/api-reference/quickstart/enrich">
    Uncover more context and insights from your documents, which Unstructured adds to your existing AI-ready JSON.
  </Card>

  <Card title="Embed" href="/api-reference/quickstart/embed">
    Power up semantic search and retrieval with vector embeddings that Unstructured generates. These
    embeddings are based on your documents' text content.
  </Card>

  <Card title="Extract" href="/api-reference/quickstart/extract">
    Extract just the data that you care about from your documents, which Unstructured outputs as JSON object data records.
  </Card>
</CardGroup>


# Transform quickstart
Source: https://docs.unstructured.io/api-reference/quickstart/transform

Create an on-demand job that uses Auto partitioning to transform local files into standard AI-ready JSON.

[Complete the requirements](/api-reference/quickstart/overview#requirements) before you begin.

You can also learn about Unstructured's [partitioning](/concepts/partitioning) before you begin.

<Steps>
  <Step title="Create the on-demand job">
    Replace `INPUT_DIR` with the path to your local directory of files to process. The response includes the job ID.

    <Note>
      Each on-demand job is limited to 10 files, and each file is limited to 50 MB in size.

      If you need to launch a series of on-demand jobs in rapid succession, you must wait at least one second between launch
      requests. Otherwise, you will receive a rate limit error.

      A maximum of 5 on-demand jobs can be running in your Unstructured account. If you launch a new on-demand job
      but 5 existing on-demand jobs are still running, the new on-demand job will remain in a scheduled state until one of the 5
      existing on-demand jobs is done running.
    </Note>

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        INPUT_DIR="/full/path/to/your/directory"

        form_args=()
        for filepath in "$INPUT_DIR"/*; do
            [ -f "$filepath" ] || continue
            filename=$(basename "$filepath")
            mimetype=$(file --mime-type -b "$filepath")
            form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
        done

        response=$(curl --request POST --location \
          "$UNSTRUCTURED_API_URL/jobs/" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --form 'request_data={"job_nodes":[{"name":"Partitioner","type":"partition","subtype":"vlm","settings":{"is_dynamic":true,"allow_fast":true}}]}' \
          "${form_args[@]}")

        JOB_ID=$(echo "$response" | jq -r '.id')
        echo "Job ID: $JOB_ID"
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import mimetypes
        import os
        import json

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import CreateJobRequest
        from unstructured_client.models.shared import BodyCreateJob, InputFiles

        INPUT_DIR = "/full/path/to/your/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        input_files = []
        for filename in os.listdir(INPUT_DIR):
            full_path = os.path.join(INPUT_DIR, filename)
            if not os.path.isfile(full_path):
                continue
            content_type, _ = mimetypes.guess_type(full_path)
            input_files.append(
                InputFiles(
                    content=open(full_path, "rb"),
                    file_name=filename,
                    content_type=content_type or "application/octet-stream"
                )
            )

        response = client.jobs.create_job(
            request=CreateJobRequest(
                body_create_job=BodyCreateJob(
                    request_data=json.dumps({
                        "job_nodes": [
                            {
                                "name": "Partitioner",
                                "type": "partition",
                                "subtype": "vlm",
                                "settings": {
                                    "is_dynamic": True,
                                    "allow_fast": True
                                }
                            }
                        ]
                    }),
                    input_files=input_files
                )
            )
        )

        job_info = response.job_information
        print(f"Job ID: {job_info.id}")
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Poll for job status">
    Replace `JOB_ID` with the job ID from the previous step. This script polls every 10 seconds and stops when the job completes.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"

        while true; do
            job=$(curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
              --header "accept: application/json" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

            status=$(echo "$job" | jq -r '.status')
            echo "Job status: $status"

            if [ "$status" = "COMPLETED" ]; then
                echo "Job completed."
                echo "Output node file IDs: $(echo "$job" | jq -c '[.output_node_files[].file_id]')"
                break
            elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
                echo "Job did not complete successfully: $status"
                exit 1
            fi

            sleep 10
        done
        ```

        This script requires [jq](https://jqlang.org/download/) to parse the JSON response.
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import os
        import time

        from unstructured_client import UnstructuredClient

        JOB_ID = "<job-id>"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        while True:
            response = client.jobs.get_job(request={"job_id": JOB_ID})
            job_info = response.job_information
            status = job_info.status

            print(f"Job status: {status.value}")

            if status == "COMPLETED":
                print("Job completed.")
                print(f"Output node file IDs: {[f.file_id for f in (job_info.output_node_files or [])]}")
                break
            elif status in ("FAILED", "STOPPED"):
                raise RuntimeError(f"Job did not complete successfully: {status}")

            time.sleep(10)
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Download the job output">
    Replace `JOB_ID`, `OUTPUT_FILE_IDS`, and `OUTPUT_DIR` with your values from the previous steps.

    <Tabs>
      <Tab title="curl">
        Save and run this script:

        ```bash theme={null}
        #!/usr/bin/env bash

        JOB_ID="<job-id>"
        OUTPUT_FILE_IDS=("<output-file-id>" "<output-file-id>") # From Step 1
        OUTPUT_DIR="/full/path/to/your/output/directory"

        mkdir -p "$OUTPUT_DIR"

        for file_id in "${OUTPUT_FILE_IDS[@]}"; do
            curl --request GET --silent --location \
              "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
              --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
              --output "$OUTPUT_DIR/$file_id.json"
            echo "Saved: $OUTPUT_DIR/$file_id.json"
        done
        ```
      </Tab>

      <Tab title="Python">
        ```python theme={null}
        import json
        import os

        from unstructured_client import UnstructuredClient
        from unstructured_client.models.operations import DownloadJobOutputRequest

        JOB_ID = "<job-id>"
        OUTPUT_FILE_IDS = ["<output-file-id>", "<output-file-id>"] # From Step 1
        OUTPUT_DIR = "/full/path/to/your/output/directory"

        client = UnstructuredClient(
            api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
            server_url=os.getenv("UNSTRUCTURED_API_URL")
        )

        os.makedirs(OUTPUT_DIR, exist_ok=True)

        for file_id in OUTPUT_FILE_IDS:
            response = client.jobs.download_job_output(
                request=DownloadJobOutputRequest(job_id=JOB_ID, file_id=file_id)
            )
            output_path = os.path.join(OUTPUT_DIR, file_id, ".json")
            with open(output_path, "w") as f:
                json.dump(response.any, f, indent=4)
            print(f"Saved: {output_path}")
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Complete end-to-end script

Replace `INPUT_DIR` and `OUTPUT_DIR` with your directory paths, then save and run this script.

<Tabs>
  <Tab title="curl">
    This script requires [jq](https://jqlang.org/download/) to parse JSON responses.

    ```bash theme={null}
    #!/usr/bin/env bash

    INPUT_DIR="/full/path/to/your/input/directory"
    OUTPUT_DIR="/full/path/to/your/output/directory"

    # Step 1: Create the on-demand job.
    form_args=()
    for filepath in "$INPUT_DIR"/*; do
        [ -f "$filepath" ] || continue
        filename=$(basename "$filepath")
        mimetype=$(file --mime-type -b "$filepath")
        form_args+=(--form "input_files=@${filepath};filename=${filename};type=${mimetype}")
    done

    response=$(curl --request POST --location \
      "$UNSTRUCTURED_API_URL/jobs/" \
      --header "accept: application/json" \
      --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
      --form 'request_data={"job_nodes":[{"name":"Partitioner","type":"partition","subtype":"vlm","settings":{"is_dynamic":true,"allow_fast":true}}]}' \
      "${form_args[@]}")

    JOB_ID=$(echo "$response" | jq -r '.id')
    echo "Job ID: $JOB_ID"

    # Step 2: Poll until the job completes.
    output_file_ids=()

    while true; do
        job=$(curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID" \
          --header "accept: application/json" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY")

        status=$(echo "$job" | jq -r '.status')
        echo "Job status: $status"

        if [ "$status" = "COMPLETED" ]; then
            echo "Job completed."

            while IFS= read -r id; do
                output_file_ids+=("$id")
            done < <(echo "$job" | jq -r '.output_node_files[].file_id')

            break
        elif [ "$status" = "FAILED" ] || [ "$status" = "STOPPED" ]; then
            echo "Job did not complete successfully: $status"
            exit 1
        fi

        sleep 10
    done

    # Step 3: Download the job output.
    mkdir -p "$OUTPUT_DIR"

    for file_id in "${output_file_ids[@]}"; do
        curl --request GET --silent --location \
          "$UNSTRUCTURED_API_URL/jobs/$JOB_ID/download?file_id=$file_id" \
          --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
          --output "$OUTPUT_DIR/$file_id.json"
        echo "Saved: $OUTPUT_DIR/$file_id.json"
    done
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    import json
    import mimetypes
    import os
    import time

    from unstructured_client import UnstructuredClient
    from unstructured_client.models.operations import CreateJobRequest, DownloadJobOutputRequest
    from unstructured_client.models.shared import BodyCreateJob, InputFiles

    INPUT_DIR = "/full/path/to/your/input/directory"
    OUTPUT_DIR = "/full/path/to/your/output/directory"

    client = UnstructuredClient(
        api_key_auth=os.getenv("UNSTRUCTURED_API_KEY"),
        server_url=os.getenv("UNSTRUCTURED_API_URL")
    )

    # Step 1: Create the on-demand job.
    input_files = []
    for filename in os.listdir(INPUT_DIR):
        full_path = os.path.join(INPUT_DIR, filename)
        if not os.path.isfile(full_path):
            continue
        content_type, _ = mimetypes.guess_type(full_path)
        input_files.append(
            InputFiles(
                content=open(full_path, "rb"),
                file_name=filename,
                content_type=content_type or "application/octet-stream"
            )
        )

    response = client.jobs.create_job(
        request=CreateJobRequest(
            body_create_job=BodyCreateJob(
                request_data=json.dumps({
                    "job_nodes": [
                        {
                            "name": "Partitioner",
                            "type": "partition",
                            "subtype": "vlm",
                            "settings": {
                                "is_dynamic": True,
                                "allow_fast": True
                            }
                        }
                    ]
                }),
                input_files=input_files
            )
        )
    )

    job_id = response.job_information.id
    print(f"Job ID: {job_id}")

    # Step 2: Poll until the job completes.
    while True:
        response = client.jobs.get_job(request={"job_id": job_id})
        job_info = response.job_information
        status = job_info.status

        print(f"Job status: {status.value}")

        if status == "COMPLETED":
            print("Job completed.")
            break
        elif status in ("FAILED", "STOPPED"):
            raise RuntimeError(f"Job did not complete successfully: {status}")

        time.sleep(10)

    output_node_file_ids = [f.file_id for f in (job_info.output_node_files or [])]

    # Step 3: Download the job output.
    os.makedirs(OUTPUT_DIR, exist_ok=True)

    for file_id in output_node_file_ids:
        response = client.jobs.download_job_output(
            request=DownloadJobOutputRequest(job_id=job_id, file_id=file_id)
        )
        output_path = os.path.join(OUTPUT_DIR, file_id, ".json")
        with open(output_path, "w") as f:
            json.dump(response.any, f, indent=4)
        print(f"Saved: {output_path}")
    ```
  </Tab>
</Tabs>

## What's next?

* Create on-demand jobs that add [enrichments](/api-reference/quickstart/enrich), add [embeddings](/api-reference/quickstart/embed), or perform [structured data extractions](/api-reference/quickstart/extract).
* Learn about the Unstructured API's other [workflow operations](/api-reference/workflow/overview).


# Supported file types
Source: https://docs.unstructured.io/api-reference/supported-file-types



The Unstructured API supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.abw`         |
| `.bmp`         |
| `.csv`         |
| `.cwk`         |
| `.dbf`         |
| `.dif`\*       |
| `.doc`         |
| `.docx`        |
| `.dot`         |
| `.dotm`        |
| `.eml`         |
| `.epub`        |
| `.et`          |
| `.eth`         |
| `.fods`        |
| `.heic`        |
| `.htm`         |
| `.html`        |
| `.hwp`         |
| `.jpeg`        |
| `.jpg`         |
| `.md`          |
| `.mcw`         |
| `.msg`         |
| `.mw`          |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pbd`         |
| `.pdf`         |
| `.png`         |
| `.pot`         |
| `.ppt`         |
| `.pptm`        |
| `.pptx`        |
| `.prn`         |
| `.rst`         |
| `.rtf`         |
| `.sdp`         |
| `.sxg`         |
| `.tif`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |
| `.zabw`        |

By file type:

| Category          | File types                                                        |
| ----------------- | ----------------------------------------------------------------- |
| Apple             | `.cwk`, `.mcw`                                                    |
| CSV               | `.csv`                                                            |
| Data Interchange  | `.dif`\*                                                          |
| dBase             | `.dbf`                                                            |
| E-mail            | `.eml`, `.msg`, `.p7s`                                            |
| EPUB              | `.epub`                                                           |
| HTML              | `.htm`, `.html`                                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.jpg`, `.png`, `.prn`, `.tif`, `.tiff` |
| Markdown          | `.md`                                                             |
| OpenOffice        | `.odt`                                                            |
| Org Mode          | `.org`                                                            |
| Other             | `.eth`, `.pbd`, `.sdp`                                            |
| PDF               | `.pdf`                                                            |
| Plain text        | `.txt`                                                            |
| PowerPoint        | `.pot`, `.ppt`, `.pptm`, `.pptx`                                  |
| reStructured Text | `.rst`                                                            |
| Rich Text         | `.rtf`                                                            |
| Spreadsheet       | `.et`, `.fods`, `.mw`, `.xls`, `.xlsx`                            |
| StarOffice        | `.sxg`                                                            |
| TSV               | `.tsv`                                                            |
| Word processing   | `.abw`, `.doc`, `.docx`, `.dot`, `.dotm`, `.hwp`, `.zabw`         |
| XML               | `.xml`                                                            |

`*` For `.dif`, `\n` characters in `.dif` files are supported, but `\r\n` characters will raise the error
`UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type`.


# Troubleshooting Unstructured API keys and URLs
Source: https://docs.unstructured.io/api-reference/troubleshooting/api-key-url



## Issue

When you run script or code to call an Unstructured API, you might get one of the following warnings or errors:

```
UserWarning: If intending to use the paid API, please define `server_url` in your request.
```

```
ERROR: Server responded with 404 - {"detail":"Not Found"}
```

You might also get the following related warnings or errors in your logs:

```
SDKError: API error occurred: Status 401
{"detail":"API key is malformed, please type the API key correctly in the header."}
```

```
API error occurred: Status 404
{"detail":"Not Found"}
```

## Possible causes

1. The Unstructured API key, API URL, or both are missing or malformed in your script or code.
2. The API key, API URL, or both are not present in your current session.
3. The API key is no longer valid, or the API key and API URL combination is not valid.

## Suggested solutions

For the API URL, this URL was provided to you when your Unstructured account was created.
If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  The default URL for the Unstructured API's workflow operations is `https://platform.unstructuredapp.io/api/v1`.

  However, you should always use the URLs that were provided to you when your Unstructured account was created.
</Note>

1. If you are not already signed in, sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

2. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account, click **API Keys** on the sidebar.<br />

   <Note>
     For other types of **Business** accounts, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

For the API URL, note the value of the Unstructured API URL that you want to call. To access these API URLs:

1. If you are not already signed in, sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

2. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to get the API URL for.
   </Note>

3. Note the API URL for the Unstructured API's workflow operations, labelled as **Unstructured API Endpoint**.

If you still believe you have the correct API URL and API key, try the following steps:

1. Check your script or code to see where the API URL and API key are specified, as follows:

   **For a POST request**

   * Check the target URL. Make sure it is set to the right API URL.
   * Check the value of the `unstructured-api-key` header. Make sure it is set to the right API key, or an environment variable representing the right API key.

   **For the Unstructured Python SDK**

   * Check the value of the `server_url` parameter. Make sure it is set to the right API URL, or an environment variable representing the right API URL.
   * Check the value of the `api_key_auth` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

   <Note>
     If you are using the Unstructured Python SDK to call the Unstructured API's workflow operations, you should not need to set the value of the
     `server_url` parameter if you are using these operations' default API URL. However, you might want to try setting this value anyway to see if it otherwise resolves your issue.
   </Note>

   **For the Unstructured JavaScript/TypeScript SDK**

   * Check the value of the `UnstructuredClient` object's `serverURL` parameter. Make sure it is set to the right API URL, or an environment variable representing the right API URL.
   * Check the value of the `UnstructuredClient` object's `security: { apiKeyAuth }` parameter. Make sure it is set to the right API key, or an environment variable representing the right API key.

2. If you follow the recommended approach to use environment variables to represent the API key and API URL instead of hard-coding them into your script or code, check your session to see if these environment variables have been loaded, for example:

   * Run the command for your operating system to list all environment variables. Make sure the API key and API URL are listed and are set to the right ones. Make sure the names and cases of these environment variables match the ones in your script or code exactly.
   * If you use an integrated development environment (IDE) to run your script or code, check the IDE's documentation for how to specify and load environment variables at run time or debug time. Make sure the names and cases of these environment variables match the ones in your script or code exactly.

3. If you are still getting this issue, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# Webhooks
Source: https://docs.unstructured.io/api-reference/webhooks



A *webhook* is a common technique that allows a *sender* to automatically send data to a *receiver* based on events that happen in the sender.
Some common uses for webhooks include sending real-time notifications such as emails, pages, or alerts to messaging apps; automatically
triggering build, test, or deployment workflows in CI/CD pipelines; launching downstream data synchronization updates; or triggering agentic AI workflows.

The following 3-minute video provides an overview of webhooks.

<iframe title="YouTube video player" />

The following 6-minute video shows how to use the Unstructured API to work with webhooks:

<iframe title="YouTube video player" />

For Unstructured webhooks, Unstructured is the sender,
and your solution is the receiver. Some popular receiver solutions include
[AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/urls-webhook-tutorial.html),
[Azure Functions](https://learn.microsoft.com/azure/azure-functions/functions-bindings-http-webhook),
[Google Cloud Run](https://docs.cloud.google.com/run/docs/triggering/webhooks),
[Zapier](https://zapier.com/blog/what-are-webhooks/),
[Slack](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/),
[Svix](https://svix.com), and
[Webhook.site](https://webhook.site/).

The receiver provides a unique URL, called the
*webhook URL*, to receive the data that Unstructured sends.
Behind the scenes, when specific events happen in Unstructured, Unstructured automatically calls the webhook URL by using an HTTP POST operation and
passes JSON payloads related to those events to the receiver. The receiver then processes the payload and decides what to do with the data.

Because webhooks are event-driven, some event must first be triggered to begin generating the related data to be sent. In
Unstructured, these webhooks can be triggered whenever a [job](/api-reference/workflow/jobs) that is associated with its target [workflow](/api-reference/workflow/workflows) does
one or more of the following:

* Is scheduled to start running later (programmatically, an event type of `job.scheduled`)
* Has begun running (`job.in_progress`)
* Has stopped running (`job.stopped`)
* Has failed to initialize  (`job.failed`) -  the job did not process any files.
* Has completed (`job.completed`) - the job initialized and finished processing. To determine whether all files were processed successfully, get the job's processing details. For more information, see [Get processing details for a job](/api-reference/api/job/get-job-details).

When a webhook is configured to deliver event-driven notification data payloads from a sender to a receiver, this configuration
is called a *notification channel*. For Unstructured webhooks, notification channels can be created and managed at the following levels:

* At the **workspace** level, known as a *workspace-scoped notification channel*. This allows any job that is associated with a
  workspace in an Unstructured account to trigger the webhook. This can be useful, for example, for routing pager requests to a team of on-call engineers in an
  IT operations center whenever a job fails across the workspace.

  <Note>
    Each **Let's Go** or **Pay-As-You-Go** account has one and only one workspace.

    An Unstructured **Business** account can have multiple workspaces.
  </Note>

* At the **workflow** level, known as a *workflow-scoped notification channel*. This allows any job that is associated only with
  the target workflow in an Unstructured account to trigger the webhook. This can be useful, for example, for emailing a department's data analyst when a long-running job for a specific workflow
  has completed, allowing them to begin working with the latest output.

Whenever a webhook is triggered, the act of Unstructured sending the event data payload is called a
*notification*. You can get a count or a list of these notifications. You can also mark them in your Unstructured account as read after
you have processed them according to your organization's needs. Marking a notification as read can help you keep track of
which notifications you have already dealt with and which ones you still need to take action on.

<Info>
  Unstructured Pipelines allows only limited creation, viewing, and management of webhooks, as follows ([learn how](/pipelines/webhooks)):

  * Webhooks for a personal Unstructured workspace. Each personal Unstructured account has one and only one personal Unstructured workspace.
  * Webhooks for a workspace within an Unstructured **Business** account.

  You cannot use Unstructured Pipelines to create, view, or manage notifications or workflow-level webhooks.
</Info>

## Verify webhook requests

Your receiver application can verify incoming webhook requests to check that the payload originated from Unstructured and was not tampered with in transit. Use the webhook secret to verify incoming requests. If you suspect that the secret has been compromised, rotate it to a new value.

The following Python example reads the webhook ID, timestamp, and signature from the request headers and checks that the timestamp is within an acceptable time window.  Next, it recomputes the expected HMAC-SHA256 signature from the webhook ID, timestamp, and raw request body using the notification channel secret. It then compares the computed signature to the versioned signature in the request header. If the values match, the request can be trusted as originating from Unstructured and as not having been tampered with in transit.

<Accordion title="Python">
  ```python theme={null}
  import hashlib
  import hmac
  import base64
  import time


  def verify_webhook(
      payload: bytes, headers: dict, secret: str, tolerance_seconds: int = 300
  ) -> bool:
      msg_id = headers["webhook-id"]
      timestamp = headers["webhook-timestamp"]
      signature_header = headers["webhook-signature"]

      # Reject stale timestamps
      if abs(time.time() - int(timestamp)) > tolerance_seconds:
          raise ValueError("Timestamp too old")

      # Use the secret as the signing key
      secret_bytes = secret.encode("utf-8")

      # Sign: "{msg_id}.{timestamp}.{raw_body}"
      signed_content = f"{msg_id}.{timestamp}.".encode() + payload
      expected = base64.b64encode(
          hmac.new(secret_bytes, signed_content, hashlib.sha256).digest()
      ).decode()

      # The header may contain multiple space-separated versioned sigs
      for sig in signature_header.split(" "):
          version, value = sig.split(",", 1)
          if version == "v1" and hmac.compare_digest(expected, value):
              return True

      raise ValueError("Invalid signature") 
  ```
</Accordion>

<Warning>
  Use the raw request body exactly as received, not a re-serialized JSON payload. Using a re-serialized payload can cause signature verification to fail.
</Warning>

## Requirements

To create, view, and manage webhooks, Unstructured provides a [set of Representational State Transfer (REST) endpoints](/api-reference/api/notification/notification-apis). You can
call these endpoints through standard REST-enabled utilities, tools, programming languages, packages, and libraries.

To call the Unstructured API's webhook operations, you must have an Unstructured API URL and a valid Unstructured API key.

To get your Unstructured API URL, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API URL:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to get the API URL for.
     [Learn more](https://docs.unstructured.io/pipelines/account/workspaces#access-a-workspace).
   </Note>

   b. Copy the value of **Unstructured API Endpoint** to your system's clipboard.

To get your Unstructured API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

<Tip>
  Each Unstructured API key works with one and only one workspace (and the workflows within that workspace). Make sure you are using the right API key
  for your target workspace!
</Tip>

For more information on workspace-level webhook operations, see [Notifications](/api-reference/api/notification/notification-apis). For more information on workflow-level webhook operations, see [Workflows](/api-reference/api/workflow/workflow-apis).


# Astra DB
Source: https://docs.unstructured.io/api-reference/workflow/destinations/astradb



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Astra DB.

## Requirements

You will need:

<iframe title="YouTube video player" />

* An IBM Cloud account or DataStax account.

  * For an IBM Cloud account, [sign up](https://cloud.ibm.com/registration) for an IBMid, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your IBMid.
  * For a DataStax account, [sign up](https://astra.datastax.com/signup) for a DataStax account, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your DataStax account.

* An Astra DB database in the DataStax account. To create a database:

  a. After you sign in to DataStax, click **Create database**.<br />
  b. Click the **Serverless (vector)** tile, if it is not already selected.<br />
  c. For **Database name**, enter some unique name for the database.<br />
  d. Select a **Provider** and a **Region**, and then click **Create database**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).

* An application token for the database. To create an application token:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Overview** tab, under **Database Details**, in the **Application Tokens** tile, click **Generate Token**.<br />
  c. Enter some **Token description** and select and **Expiration** time period, and then click **Generate token**.<br />
  d. Save the application token that is displayed to a secure location, and then click **Close**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).

* A keyspace in the database. To create a keyspace:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select **Create keyspace**.<br />
  c. Enter some **Keyspace name**, and then click **Add keyspace**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-keyspaces.html#keyspaces).

* A collection in the keyspace.

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * An existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. If Unstructured generates embeddings,
      the new collection's name will be `u<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `u<short-workflow-id`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * For the source connector only, an existing collection is required.
  * For the destination connector only, an existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. The new collection's name will be `unstructuredautocreated`.

  To create a collection yourself:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select the name of the target keyspace.<br />
  c. In the **Collections** list, select **Create collection**.<br />
  d. Enter some **Collection name**.<br />
  e. Turn on **Vector-enabled collection**, if it is not already turned on.<br />
  f. Choose a mode for **Embedding generation method**. See [Astra DB generated embeddings](#astra-db-generated-embeddings).<br />
  g. If you chose **Bring my own**, enter the number of dimensions for the embedding model that you plan to use.<br />
  h. For **Similarity metric**, select **Cosine**.<br />
  i. Click **Create collection**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

## Examples

To create an Astra DB destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="astradb",
                  config={
                      "token": "<token>",
                      "api_endpoint": "<api-endpoint>",
                      "collection_name": "<collection-name>",
                      "keyspace": "<keyspace>",
                      "batch_size": <batch-size>,
                      "flatten_metadata": <True|False>,
                      "binary_encode_vectors": <True|False>,
                      "enable_lexical_search": <True|False>,
                      "astra_generated_embeddings": <True|False>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "astradb",
      "config": {
          "token": "<token>",
          "api_endpoint": "<api-endpoint>",
          "collection_name": "<collection-name>",
          "keyspace": "<keyspace>",
          "batch_size": <batch-size>,
          "flatten_metadata": "<true|false>",
          "binary_encode_vectors": "<true|false>",
          "enable_lexical_search": "<true|false>",
          "astra_generated_embeddings": "<true|false>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The application token for the database.
</ParamField>

<ParamField type="string">
  The database's associated API endpoint.
</ParamField>

<ParamField type="string">
  The name of the collection in the keyspace. If no value is provided, see the beginning of this article for the behavior at run time.
</ParamField>

<ParamField type="string">
  The name of the keyspace in the collection.
</ParamField>

<ParamField type="integer">
  The maximum number of records to send per batch.
</ParamField>

<ParamField type="boolean">
  Set to `true` to flatten the metadata into each record. Specifically, when flattened, the metadata key values are brought to the top level of the element, and the `metadata` key itself is removed.
</ParamField>

<ParamField type="boolean">
  Set to `false` to upload vectors as a list of numbers. This is more efficient, but it makes it harder to view and work with vectors in the UI.
</ParamField>

<ParamField type="boolean">
  Set to `true` to enable lexical and hybrid search. See [Lexical search](#lexical-search).
</ParamField>

<ParamField type="boolean">
  Set to `true` to use an Astra vectorize integration for embeddings. See [Astra DB generated embeddings](#astra-db-generated-embeddings).
</ParamField>

## Lexical search

When **Enable Lexical Search** is enabled, document text will be inserted into the `$lexical` field for lexical and hybrid search capabilities.

<Note>
  Your collection must be configured for lexical search, and your database must be in one of the supported AWS regions. For more information, see [Find data with lexicographical matching](https://docs.datastax.com/en/astra-db-serverless/databases/lexical-search.html) in the Astra DB documentation. Otherwise, you will encounter the following error when running a job:

  ` The Collection <keyspace>.<collection> does not have Lexical feature enabled. (LEXICAL_NOT_ENABLED_FOR_COLLECTION) [with 0 inserted ids])`
</Note>

## Astra DB generated embeddings

When **Astra DB Generated Embeddings** is enabled:

* Your document text will be inserted into the `$vectorize` field.
* Astra DB will automatically populate the `$vector` field with embeddings.
* Your workflow **must not** include an embedder node.
* You **must** have an embedding provider configured for your collection.

When **Astra DB Generated Embeddings** is disabled (the default):

* Unstructured will insert embeddings directly into the `$vector` field
* Your workflow **must** include an embedder node.

For more information, see [Auto-generate embeddings with vectorize](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html) in the Astra DB documentation.

### Troubleshooting

You might encounter the following errors when running a job that uses this connector:

**Error**:

`Unable to vectorize data, embedding service not configured for the collection : <collection> (EMBEDDING_SERVICE_NOT_CONFIGURED) [with 0 inserted ids])`

You've enabled **Astra DB Generated Embeddings**, but your collection does not have an embedding provider. Either disable **Astra DB Generated Embeddings** in your connector settings, or recreate your collection with an embedding provider.

**Error**:

`Cannot use Unstructured embeddings and Astra-generated embeddings simultaneously. Please disable Astra generated embeddings or remove the Unstructured embedder.`

**or**:

`No vectors provided. Please enable an Unstructured embedding provider or configure Astra to generate embeddings.`

You must choose either Unstructured or Astra DB embeddings. Configuring neither, or both, is unsupported.


# Azure AI Search
Source: https://docs.unstructured.io/api-reference/workflow/destinations/azure-ai-search



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Azure AI Search.

## Requirements

You will need:

The following video shows how to fulfill the minimum set of Azure AI Search requirements:

<iframe title="YouTube video player" />

Here are some more details about these requirements:

* The endpoint and API key for Azure AI Search. [Create an endpoint and API key](https://learn.microsoft.com/azure/search/search-create-service-portal).
* The name of the index in Azure AI Search. [Create an index](https://learn.microsoft.com/rest/api/searchservice/create-index).

  <iframe title="YouTube video player" />

  The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents
  that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  You can adapt the following index schema example for your own needs. Be sure to replace `<number-of-dimensions>`
  (in three locations in the following example) with the number of dimensions of the embedding model you are using:

  ```json theme={null}
  {
    "name": "elements-index",
    "fields": [
      {
        "name": "id",
        "type": "Edm.String",
        "key": true
      },
      {
        "name": "record_id",
        "type": "Edm.String",
        "filterable": true
      },
      {
        "name": "element_id",
        "type": "Edm.String"
      },
      {
        "name": "text",
        "type": "Edm.String",
        "searchable": true
      },
      {
        "name": "type",
        "type": "Edm.String"
      },
      {
        "name": "metadata",
        "type": "Edm.ComplexType",
        "fields": [
          {
            "name": "orig_elements",
            "type": "Edm.String"
          },
          {
            "name": "category_depth",
            "type": "Edm.Int32"
          },
          {
            "name": "parent_id",
            "type": "Edm.String"
          },
          {
            "name": "attached_to_filename",
            "type": "Edm.String"
          },
          {
            "name": "filetype",
            "type": "Edm.String"
          },
          {
            "name": "last_modified",
            "type": "Edm.DateTimeOffset"
          },
          {
            "name": "is_continuation",
            "type": "Edm.Boolean"
          },
          {
            "name": "file_directory",
            "type": "Edm.String"
          },
          {
            "name": "filename",
            "type": "Edm.String"
          },
          {
            "name": "data_source",
            "type": "Edm.ComplexType",
            "fields": [
              {
                "name": "url",
                "type": "Edm.String"
              },
              {
                "name": "version",
                "type": "Edm.String"
              },
              {
                "name": "date_created",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "date_modified",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "date_processed",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "permissions_data",
                "type": "Edm.String"
              },
              {
                "name": "record_locator",
                "type": "Edm.String"
              }
            ]
          },
          {
            "name": "coordinates",
            "type": "Edm.ComplexType",
            "fields": [
              {
                "name": "system",
                "type": "Edm.String"
              },
              {
                "name": "layout_width",
                "type": "Edm.Double"
              },
              {
                "name": "layout_height",
                "type": "Edm.Double"
              },
              {
                "name": "points",
                "type": "Edm.String"
              }
            ]
          },
          {
            "name": "languages",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "page_number",
            "type": "Edm.String"
          },
          {
            "name": "links",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "page_name",
            "type": "Edm.String"
          },
          {
            "name": "link_urls",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "link_texts",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "sent_from",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "sent_to",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "subject",
            "type": "Edm.String"
          },
          {
            "name": "section",
            "type": "Edm.String"
          },
          {
            "name": "header_footer_type",
            "type": "Edm.String"
          },
          {
            "name": "emphasized_text_contents",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "emphasized_text_tags",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "text_as_html",
            "type": "Edm.String"
          },
          {
            "name": "regex_metadata",
            "type": "Edm.String"
          },
          {
            "name": "detection_class_prob",
            "type": "Edm.Double"
          }
        ]
      },
      {
        "name": "embeddings",
        "type": "Collection(Edm.Single)",
        "dimensions": <number-of-dimensions>,
        "vectorSearchProfile": "embeddings-config-profile"
      }
    ],
    "vectorSearch": {
      "algorithms": [
        {
          "name": "hnsw-<number-of-dimensions>",
          "kind": "hnsw",
          "hnswParameters": {
            "m": 4,
            "efConstruction": 400,
            "efSearch": 500,
            "metric": "cosine"
          }
        }
      ],
      "profiles": [
        {
          "name": "embeddings-config-profile",
          "algorithm": "hnsw-<number-of-dimensions>"
        }
      ]
    },
    "semantic": {
      "configurations": [
        {
          "name": "default-semantic-config",
          "prioritizedFields": {
            "titleField": null,
            "prioritizedContentFields": [
              { "fieldName": "text" }
            ],
            "prioritizedKeywordsFields": []
          }
        }
      ]
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [Search indexes in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-an-index)
  * [Schema of a search index](https://learn.microsoft.com/azure/search/search-what-is-an-index#schema-of-a-search-index)
  * [Example index schema](https://learn.microsoft.com/rest/api/searchservice/create-index#examples)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

## Examples

The following examples create an Azure AI Search destination connector using the Unstructured API.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="azure_ai_search",
                  config={
                      "endpoint": "<endpoint>",
                      "index": "<index>",
                      "key": "<key>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "azure_ai_search",
      "config": {
          "endpoint": "<endpoint>",
          "index": "<index>",
          "key": "<key>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The endpoint URL for your Azure AI Search service, in the format `https://<service-name>.search.windows.net`.
</ParamField>

<ParamField type="string">
  The name of the index in your Azure AI Search service.
</ParamField>

<ParamField type="string">
  The admin API key for your Azure AI Search service. Required if not using Enterprise Connect authentication.
</ParamField>

<h2>
  Set up Enterprise Connect authentication
</h2>

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see [Enterprise Connect for Azure](/business/azure/enterprise-connect).

To configure an Azure AI Search connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

1. Enable role-based access on your Azure AI Search service.

   <Warning>
     Azure AI Search defaults to API key authentication only. Complete this step to configure the service to accept the Microsoft Entra ID tokens that Enterprise Connect uses.
   </Warning>

   Follow the instructions in [Enable role-based access control for Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-security-enable-roles) in the Azure AI Search documentation. In the Azure portal, navigate to your search service, select **Settings** > **Keys**, and select **Both** to allow both API key and role-based authentication. If you want to use role-based authentication exclusively, select **Role-based access control** instead.

2. Create an App Registration for Unstructured in Microsoft Entra ID.

   In your Azure subscription, follow the instructions in [How to register an app in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, `unstructured-connector`). For **Supported account types**, select **Single tenant only**.

   You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the [single-tenant scenario](https://learn.microsoft.com/en-us/entra/identity-platform/single-and-multi-tenant-apps) as defined by Microsoft.

3. Add a federated identity credential to the App Registration.

   Follow the instructions in [Configure an app to trust an external identity provider](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust) in the Microsoft Entra documentation. Navigate to your App Registration, select **Certificates & secrets** in the left navigation pane, select the **Federated credentials** tab, and select **Add credential**.

   For **Federated credential scenario**, select **Other issuer**.

   Set the following values:

   | Field        | Value                                                                                                                                                                                                                                                                                                            |
   | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**     | A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.                                                                                                                                                                                     |
   | **Issuer**   | The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)                                                                                                                                                       |
   | **Subject**  | The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error. |
   | **Audience** | Set this to `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                                        |

   Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

4. Add a role assignment to grant your App Registration access to your Azure AI Search service.

   See [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) in the Azure documentation. Use the following values:

   * **Scope**: the Azure AI Search service that contains the index you want the connector to access.
   * **Role**: assign both **Search Index Data Contributor** (required for document indexing) and **Search Service Contributor** (required for connection testing). Repeat the role assignment steps for each role.
   * **Members**: select **User, group, or service principal**, then search for and select the App Registration you created in Step 2.

   When you reach the **Review + assign** tab, click **Review + assign** to complete the assignment.

5. Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the **Overview** page of your App Registration in the [Microsoft Entra admin center](https://entra.microsoft.com).

   * The **Tenant ID** (also called Directory ID) for your Azure subscription.
   * The **Client ID** of your App Registration.

Next, see the **Create the destination connector with Enterprise Connect** section below for examples.

### Create the destination connector with Enterprise Connect

The following examples create an Azure AI Search destination connector using Enterprise Connect authentication.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="azure_ai_search",
                  config={
                      "endpoint": "<endpoint>",
                      "index": "<index>",
                      "tenant_id": "<tenant-id>",
                      "client_id": "<client-id>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "azure_ai_search",
      "config": {
          "endpoint": "<endpoint>",
          "index": "<index>",
          "tenant_id": "<tenant-id>",
          "client_id": "<client-id>"
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The endpoint URL for your Azure AI Search service, in the format `https://<service-name>.search.windows.net`.
</ParamField>

<ParamField type="string">
  The name of the index in your Azure AI Search service.
</ParamField>

<ParamField type="string">
  The Tenant ID (also called Directory ID) for your Azure subscription.
</ParamField>

<ParamField type="string">
  The Client ID of your App Registration.
</ParamField>


# Azure Blob Storage
Source: https://docs.unstructured.io/api-reference/workflow/destinations/azure-blob-storage



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Azure Blob Storage.

## Requirements

You will need:

The following video shows how to fulfill the minimum set of Azure Storage account requirements:

<iframe title="YouTube video player" />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  * **Read** and **List** for reading from the container only.
  * **Write** and **List** for writing to the container only.
  * **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these requirements:

* An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

  <iframe title="YouTube video player" />

* An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

  <iframe title="YouTube video player" />

  <iframe title="YouTube video player" />

* The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

  Create an SAS token (recommended):

  <iframe title="YouTube video player" />

  Get an access key or connection string:

  <iframe title="YouTube video player" />

## Examples

To create an Azure Blob Storage destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="azure",
                  config={
                      "remote_url": "az://<container-name>/<path/to/file/or/folder>",
                      
                      # For SAS token authentication:
                      # "account_name": "<account-name>",
                      # "sas_token": "<sas-token>",

                      # For account key authentication:
                      # "account_name": "<account-name>",
                      # "account_key": "<account-key>",

                      # For connection string authentication:
                      # "connection_string": "<connection-string>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "azure",
      "config": {
          "remote_url": "az://<container-name>/<path/to/file/or/folder>",
          
          # For SAS token authentication:
          "account_name": "<account-name>",
          "sas_token": "<sas-token>"

          # For account key authentication:
          "account_name": "<account-name>",
          "account_key": "<account-key>"

          # For connection string authentication:
          "connection_string": "<connection-string>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`. For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the Azure Storage remote URL would be `az://my-container/my-folder/`.
</ParamField>

<ParamField type="string">
  The Azure Storage account name. Required for SAS token authentication and account key authentication.
</ParamField>

<ParamField type="string">
  For SAS token authentication, the SAS token for the Azure Storage account.
</ParamField>

<ParamField type="string">
  For account key authentication, the key for the Azure Storage account.
</ParamField>

<ParamField type="string">
  For connection string authentication, the connection string for the Azure Storage account.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to recursively access files from subfolders within the container.
</ParamField>

<h2>
  Set up Enterprise Connect authentication
</h2>

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see [Enterprise Connect for Azure](/business/azure/enterprise-connect).

To configure an Azure Blob Storage connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

1. Create an App Registration for Unstructured in Microsoft Entra ID.

   In your Azure subscription, follow the instructions in [How to register an app in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, `unstructured-connector`). For **Supported account types**, select **Single tenant only**.

   You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the [single-tenant scenario](https://learn.microsoft.com/en-us/entra/identity-platform/single-and-multi-tenant-apps) as defined by Microsoft.

2. Add a federated identity credential to the App Registration.

   Follow the instructions in [Configure an app to trust an external identity provider](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust) in the Microsoft Entra documentation. Navigate to your App Registration, select **Certificates & secrets** in the left navigation pane, select the **Federated credentials** tab, and select **Add credential**.

   For **Federated credential scenario**, select **Other issuer**.

   Set the following values:

   | Field        | Value                                                                                                                                                                                                                                                                                                            |
   | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**     | A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.                                                                                                                                                                                     |
   | **Issuer**   | The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)                                                                                                                                                       |
   | **Subject**  | The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error. |
   | **Audience** | Set this to `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                                        |

   Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

3. Add a role assignment to grant your App Registration access to your Azure Blob Storage account.

   See [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) in the Azure documentation. Use the following values:

   * **Scope**: the Azure Blob Storage account that contains the data you want the connector to access.
   * **Role**: select **Storage Blob Data Reader** for a source, or **Storage Blob Data Contributor** for a destination.
   * **Members**: select **User, group, or service principal**, then search for and select the App Registration you created in Step 1.

   When you reach the **Review + assign** tab, click **Review + assign** to complete the assignment.

4. Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the **Overview** page of your App Registration in the [Microsoft Entra admin center](https://entra.microsoft.com).

   * The **Tenant ID** (also called Directory ID) for your Azure subscription.
   * The **Client ID** of your App Registration.

Next, see the **Create the connector with Enterprise Connect** section below for examples.

### Create the destination connector with Enterprise Connect

The following examples create an Azure Blob Storage destination connector using Enterprise Connect authentication.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="azure",
                  config={
                      "remote_url": "az://<container-name>/<path/to/file/or/folder>",
                      "account_name": "<account-name>",
                      "tenant_id": "<tenant-id>",
                      "client_id": "<client-id>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "azure",
      "config": {
          "remote_url": "az://<container-name>/<path/to/file/or/folder>",
          "account_name": "<account-name>",
          "tenant_id": "<tenant-id>",
          "client_id": "<client-id>"
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`. For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the Azure Storage remote URL would be `az://my-container/my-folder/`.
</ParamField>

<ParamField type="string">
  The Azure Storage account name.
</ParamField>

<ParamField type="string">
  The Tenant ID (also called Directory ID) for your Azure subscription.
</ParamField>

<ParamField type="string">
  The Client ID of your App Registration.
</ParamField>


# Couchbase
Source: https://docs.unstructured.io/api-reference/workflow/destinations/couchbase



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Couchbase.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Couchbase Capella clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe title="YouTube video player" />

For Couchbase Capella, you will need:

* A [Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier).
* A [Couchbase Capella cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started).
* A [bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket),
  [scope](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-scope),
  and [collection](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-collection)
  on the cluster.
* The cluster's [public connection string](https://docs.couchbase.com/cloud/get-started/connect.html#connect-from-sdk-cbsh-cli-or-ide).
* The [cluster access name (username) and secret (password)](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials).
* [Incoming IP address allowance](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) for the cluster.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

For a local Couchbase server, you will need:

* [Installation of a local Couchbase server](https://docs.couchbase.com/server/current/getting-started/start-here.html).
* [Connection details](https://docs.couchbase.com/server/current/guides/connect.html) to the local Couchbase server.

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

## Examples

To create a Couchbase destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="couchbase",
                  config={
                      "username": "<username>",
                      "bucket": "<bucket>",
                      "connection_string": "<connection-string>",
                      "scope": "<scope>",
                      "collection": "<collection>",
                      "password": "<password>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  # ...
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "couchbase",
      "config": {
          "username": "<username>",
          "bucket": "<bucket>",
          "connection_string": "<connection-string>",
          "scope": "<scope>",
          "collection": "<collection>",
          "password": "<password>",
          "batch_size": <batch-size>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The username for the Couchbase server.
</ParamField>

<ParamField type="string">
  The name of the bucket in the Couchbase server.
</ParamField>

<ParamField type="string">
  The connection string for the Couchbase server.
</ParamField>

<ParamField type="string">
  The name of the scope in the bucket.
</ParamField>

<ParamField type="string">
  The name of the collection in the scope.
</ParamField>

<ParamField type="string">
  The password for the Couchbase server.
</ParamField>

<ParamField type="integer">
  The maximum number of records to transmit per batch.
</ParamField>

<ParamField type="string">
  Source connector only. The name of the collection field that contains the document ID.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Couchbase Integration in Unstructured Platform](https://unstructured.io/blog/couchbase-integration-in-unstructured-platform)


# Delta Tables in Databricks
Source: https://docs.unstructured.io/api-reference/workflow/destinations/databricks-delta-table

Send processed data from Unstructured to a Delta Table in Databricks.

<Tip>
  This article covers connecting Unstructured to Delta Tables in Databricks.

  For information about connecting Unstructured to Delta Tables in Amazon S3 instead, see
  [Delta Tables in Amazon S3](/api-reference/workflow/destinations/delta-table).

  For information about connecting Unstructured to Databricks Volumes instead, see
  [Databricks Volumes](/api-reference/workflow/destinations/databricks-volumes).
</Tip>

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* [Account and resources](#accounts-and-resources)
* [Unity Catalog resources](#unity-catalog-resources)
* [Authentication and permissions](#authentication-and-permissions)

### Accounts and resources

* A Databricks account on [Amazon Web Services (AWS)](https://docs.databricks.com/getting-started/free-trial.html), [Microsoft Azure (Azure)](https://learn.microsoft.com/azure/databricks/getting-started/), or [Google Cloud Platform (GCP)](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html), [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* One of the following compute resources within the workspace:

  * A SQL warehouse for [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html), [Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create), or [GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html).

    The following video shows how to create a SQL warehouse if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the warehouse's owner to use it:

    <iframe title="YouTube video player" />

  * An all-purpose cluster for [AWS](https://docs.databricks.com/compute/use-compute.html), [Azure](https://learn.microsoft.com/azure/databricks/compute/use-compute), or [GCP](https://docs.gcp.databricks.com/compute/use-compute.html).

    The following video shows how to create an all-purpose cluster if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the cluster's owner to use it:

    <iframe title="YouTube video player" />

* The SQL warehouse's or cluster's **Server Hostname** and **HTTP Path** values for [AWS](https://docs.databricks.com/integrations/compute-details.html),
  [Azure](https://learn.microsoft.com/azure/databricks/integrations/compute-details), or
  [GCP](https://docs.gcp.databricks.com/integrations/compute-details.html).

* Unity Catalog enabled in the workspace for [AWS](https://docs.databricks.com/data-governance/unity-catalog/get-started.html),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/get-started), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/get-started.html).

### Unity Catalog resources

Within Unity Catalog:

* A catalog for [AWS](https://docs.databricks.com/catalogs/create-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html).
* A schema for [AWS](https://docs.databricks.com/schemas/create-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) within that catalog.

  <Note>
    A *schema* was formerly known as a *database* in Unity Catalog.
  </Note>
* A table for [AWS](https://docs.databricks.com/tables/managed.html), [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or [GCP](https://docs.gcp.databricks.com/tables/managed.html) within that schema.

  How you create and specify the table depends on whether you want document metadata stored as nested JSON in a single column, or have Unstructured flatten the metadata and store each field as a separate columns. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.

  * If you choose to store document metadata as a JSON blob, you can specify a table, or have the connector attempt to create a table for you automatically at run time.

    ***To specify an existing table***

    In the connector settings as described later in this article, specify the name of an existing table.

    The following video shows how to create a catalog, schema, and a table in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

    <iframe title="YouTube video player" />

    If you want to use an existing table or create one yourself beforehand, this table must contain at minimum the following column names and their data types:

    ```text theme={null}
    CREATE TABLE IF NOT EXISTS <catalog_name>.<schema_name>.<table_name> (
        id STRING NOT NULL PRIMARY KEY,
        record_id STRING NOT NULL,
        element_id STRING NOT NULL,
        text STRING,
        embeddings ARRAY<FLOAT>,
        type STRING,
        metadata VARIANT
    );
    ```

    <Info>
      In Databricks, a table's *schema* is different than a *schema* in a catalog-schema object relationship in Unity Catalog.
    </Info>

    <Info>
      The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
    </Info>

    ***To have the connector create a table***

    You can have the connector attempt to create a table for you automatically at run time. In the connector settings as described later in this article, do one of the following:

    * Specify the name of the table that you want the connector to attempt to create within the specified catalog and schema.
    * Leave the table name blank. The connector will attempt to create a table within the specified catalog and schema.

      For [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview), the table is named `u<short-workflow-id>`.
      For the [Unstructured Ingest CLI and Ingest Python library](/open-source/ingestion/overview), the table is named `unstructuredautocreated`.

    The connector will attempt to create the table on behalf of the related Databricks workspace user or Databricks managed service principal that is referenced in the connector settings, as described later in these requirements.
    If successful, the table's owner is set as the related Databricks workspace user or Databricks managed service principal. The owner will have all Unity Catalog
    privileges on the table by default. No other Databricks workspace users or Databricks managed service principals will have any privileges on the table by default.

    <Warning>
      If the table's parent schema is not owned by the same Databricks workspace user or Databricks managed service principal that is
      referenced in the connector settings, then you should grant the new table's owner the `CREATE TABLE` privilege on that parent schema
      before the connector attempts to create the table. Otherwise, table creation could fail.
    </Warning>

    <Note>
      Using dashes (`-`) in the names of catalogs, schemas, and tables might cause isolated issues with the connector. It is
      recommended to use underscores (`_`) instead of dashes in the names of catalogs, schemas, and tables.
    </Note>

  * If you choose to flatten the document metadata, you must create the table before you configure your connector. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.

### Authentication and permissions

* For Databricks managed service principal authentication (using Databricks OAuth M2M) to the workspace:

  * A Databricks managed service principal.
    This service principal must have the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse.
  * The service principal's **UUID** (or **Client ID** or **Application ID**) value.
  * The OAuth **Secret** value for the service principal.

  To get this information, see Steps 1-3 of the instructions for [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
  [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

  <Note>
    For Azure Databricks, this connector only supports Databricks managed service principals for authentication.
    Microsoft Entra ID managed service principals are not supported.
  </Note>

  The following video shows how to create a Databricks managed service principal if you do not already have one available:

  <iframe title="YouTube video player" />

* For Databricks personal access token authentication to the workspace, the
  Databricks personal access token value for
  [AWS](https://docs.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat#azure-databricks-personal-access-tokens-for-workspace-users), or
  [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users).
  This token must be for the workspace user who
  has the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse,

  The following video shows how to create a Databricks personal access token if you do not already have one available:

  <iframe title="YouTube video player" />

* The Databricks workspace user or Databricks managed service principal must have the following *minimum* set of permissions and privileges to write to an
  existing volume or table in Unity Catalog. If the owner of these is that Databricks workspace user or Databricks managed service principal, then
  they will have all necessary permissions and privileges by default. If the owner is someone else, then the following permissions and privileges must be
  explicitly granted to them before using the connector:

  * To use an all-purpose cluster for access, `Can Restart` permission on that cluster. Learn how to check and set cluster permissions for
    [AWS](https://docs.databricks.com/compute/clusters-manage.html#compute-permissions),
    [Azure](https://learn.microsoft.com/azure/databricks/compute/clusters-manage#cluster-level-permissions), or
    [GCP](https://docs.gcp.databricks.com/compute/clusters-manage.html#compute-permissions).

  * To use a SQL warehouse for access, `Can use` permission on that SQL warehouse. Learn how to check and set SQL warehouse permissions for
    [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html#manage-a-sql-warehouse),
    [Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create#manage), or
    [GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html#manage-a-sql-warehouse).

  * To access a Unity Catalog volume, the following privileges:

    * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
    * `USE SCHEMA` on the volume's parent schema in Unity Catalog.
    * `READ VOLUME` and `WRITE VOLUME` on the volume.

    Learn how to check and set Unity Catalog privileges for
    [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
    [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
    [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

  * To access a Unity Catalog table, the following privileges:

    * `USE CATALOG` on the table's parent catalog in Unity Catalog.
    * `USE SCHEMA` on the table's parent schema in Unity Catalog.
    * To create a new table, `CREATE TABLE` on the table's parent schema in Unity Catalog.
    * If the table already exists, `MODIFY` and `SELECT` on the table.

    Learn how to check and set Unity Catalog privileges for
    [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
    [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
    [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

* Within Unity Catalog, a volume
  for [AWS](https://docs.databricks.com/volumes/utility-commands.html),
  [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands),
  or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html). The volume can be in the same
  schema as the table, or the volume and table can be in separate schemas. In either case, both of these
  schemas must share the same parent catalog.

  <Note>
    Using dashes (`-`) in the names of volumes might cause isolated issues with the connector. It is
    recommended to use underscores (`_`) instead of dashes in the names of volumes.
  </Note>

  The following video shows how to create a catalog, schema, and a volume in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

  <iframe title="YouTube video player" />

  ## Storing document metadata

  Unstructured offers the following options for storing [document metadata](/concepts/document-elements#metadata) in the destination table:

  * Store the metadata as a single nested JSON field.
  * *Flatten* the metadata by writing each metadata field as its own typed, queryable column.

  Each of these approaches has its advantages. In general, storing the metadata as a JSON blob works for most use cases, including when:

  * Performing dot.notation queries on the stored JSON is sufficient for your needs.
  * Document metadata schemas vary across file sources. When flattening document metadata, Unstructured drops fields that do not match existing columns in the schema.
  * You want the connector to automatically generate the destination table. This option is not supported when flattening document metadata.

  Flattening the metadata and storing it in multiple columns is most useful when:

  * You want to query individual metadata fields directly using standard SQL, without JSON parsing.
  * The business intelligence or analytics tools you are using require columnar data.

  Storing metadata as a JSON blob is the default.

  To store metadata as a JSON blob, when configuring the connector uncheck **Flatten Metadata** (in the Unstructured Pipelines), or set `flatten_metadata` to `false` (in the Unstructured API).  To flatten the metadata, check **Flatten Metadata**, or set `flatten_metadata` to `true`.

  ### How Unstructured flattens metadata

  If you choose to have Unstructured flatten document metadata, you must create the Delta Table to use as the destination, and specify it when creating the connector. In order to prevent possible data loss, Unstructured will not automatically create a new table.

  Considerations to keep in mind when creating the Delta Table:

  * The table must contain a column for each metadata field you want to store. Any metadata field that does not have a corresponding column in the table is silently dropped, although the event is written to the logs. For more information, see [Logging and monitoring](/business/security-compliance/overview#logging-and-monitoring).
  * Do not declare metadata columns as `NOT NULL`. Missing metadata values are written as `NULL`.
  * Unstructured passes values through as their JSON-native type: strings, numbers, boolean, and so on. For example, no special formatting is applied to timestamp values.
  * Metadata fields that are lists are not further flattened. Lists remain single columns.

  ### Metadata flattening example

  The following example demonstrates how Unstructured flattens metadata into separate columns. Consider the following metadata:

  ```json theme={null}
  {
    "metadata": {
      "category_depth": 1,
      "data_source": {
        "url": "s3://my-source-bucket/path/chart-and-table.pdf",
        "version": "864ae40b0719e976e98ba0a7b9fcba92",
        "record_locator": {
          "protocol": "s3",
          "remote_file_path": "s3://my-source-bucket/path/"
        }
      },
      "languages": ["eng"]
    }
  }
  ```

  When flattening metadata, Unstructured generates a field name comprised of the full path to that field within the metadata structure, from the outermost object to the field itself. For example, `protocol`, which is included in the `record_locator` object, which is in turn within `data_source`, becomes `data_source_record_locator_protocol`:

  ```json theme={null}
  {
    "category_depth": 1,
    "data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
    "data_source_version": "864ae40b0719e976e98ba0a7b9fcba92",
    "data_source_record_locator_protocol": "s3",
    "data_source_record_locator_remote_file_path": "s3://my-source-bucket/path/",
    "languages": ["eng"]
  }
  ```

## Examples

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="databricks_volume_delta_tables",
                  config={
                      "server_hostname": "<server-hostname>",
                      "http_path": "<http-path>",
                      "token": "<token>",
                      "client_id": "<client-id>",
                      "client_secret": "<client-secret>",
                      "volume": "<volume>",
                      "catalog": "<catalog>",
                      "volume_path": "<volume_path>",
                      "schema": "<schema>",
                      "database": "<database>",
                      "table_name": "<table_name>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "databricks_volume_delta_tables",
      "config": {
          "server_hostname": "<server-hostname>",
          "http_path": "<http-path>",
          "token": "<token>",
          "client_id": "<client-id>",
          "client_secret": "<client-secret>",
          "volume": "<volume>",
          "catalog": "<catalog>",
          "volume_path": "<volume_path>",
          "schema": "<schema>",
          "database": "<database>",
          "table_name": "<table_name>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The target Databricks cluster's or SQL warehouse's **Server Hostname** value.
</ParamField>

<ParamField type="string">
  The cluster's or SQL warehouse's **HTTP Path** value.
</ParamField>

<ParamField type="string">
  For Databricks personal access token (PAT) authentication, the target Databricks user's PAT value. Required for PAT authentication.
</ParamField>

<ParamField type="string">
  For Databricks OAuth machine-to-machine (M2M) authentication, the Databricks managed service principal's **UUID** (or **Client ID** or **Application ID**) value. Required for OAuth authentication.
</ParamField>

<ParamField type="string">
  For Databricks OAuth M2M authentication, the Databricks managed service principal's OAuth **Secret** (client secret) value. Required for OAuth authentication.
</ParamField>

<ParamField type="string">
  The name of the catalog in Unity Catalog for the target volume and table in the Databricks workspace.
</ParamField>

<ParamField type="string">
  The name of the schema (formerly known as a database) in Unity Catalog for the target table. If the target table and volume are in the same schema, then `database` and `schema` will have the same values.
</ParamField>

<ParamField type="string">
  The name of the target table in Unity Catalog. If specified but the table does not exist, the connector attempts to create it. If not specified, the connector attempts to create a table named `u<short-workflow-id>`. See the beginning of this article for additional technical requirements.
</ParamField>

<ParamField type="string">
  The name of the schema (formerly known as a database) in Unity Catalog for the target volume. If the target volume and table are in the same schema, then `schema` and `database` will have the same values.
</ParamField>

<ParamField type="string">
  The name of the target volume in Unity Catalog.
</ParamField>

<ParamField type="string">
  Any target folder path inside of the volume to use instead of the volume's root. If not otherwise specified, processing occurs at the volume's root.
</ParamField>

<Note>
  Using dashes (`-`) in the names of catalogs, schemas (formerly known as databases), tables, and volumes might cause isolated issues with the connector. It is
  recommended to use underscores (`_`) instead of dashes in the names of catalogs, schemas, tables, and volumes.
</Note>

<ParamField type="boolean">
  Set to `true` to have Unstructured flatten the metadata and store each field as a separate columns, or `false` to store document metadata as nested JSON in a single column. For more information, see [Storing document metadata](#storing-document-metadata).
</ParamField>

## Learn more

* <Icon icon="blog" />  [Integration Highlight: Databricks Delta Tables](https://unstructured.io/blog/integration-highlight-databricks-delta-tables)


# Databricks Volumes
Source: https://docs.unstructured.io/api-reference/workflow/destinations/databricks-volumes



<Tip>
  This article covers connecting Unstructured to Databricks Volumes.

  For information about connecting Unstructured to Delta Tables in Databricks instead, see
  [Delta Tables in Databricks](/api-reference/workflow/destinations/databricks-delta-table).
</Tip>

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Databricks Volumes.

## Requirements

You will need:

* A Databricks account on [AWS](https://docs.databricks.com/getting-started/free-trial.html),
  [Azure](https://learn.microsoft.com/azure/databricks/getting-started/), or
  [GCP](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or
  [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* The workspace's URL. Get the workspace URL for
  [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
  [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
  or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

  Examples:

  * AWS: `https://<workspace-id>.cloud.databricks.com`
  * Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
  * GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

  <Note>
    Do not add a trailing slash (`/`) to the workspace URL.
  </Note>

* The Databricks authentication details. For more information, see the documentation for
  [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
  or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

  For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), the following Databricks authentication types are supported:

  * Databricks OAuth machine-to-machine (M2M) authentication for\
    [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

    You will need the **Client ID** (or **UUID** or **Application** ID) and OAuth **Secret** (client secret) values for the corresponding service principal.
    Note that for Azure, only Databricks managed service principals are supported. Microsoft Entra ID managed service principals are not supported.

    The following video shows how to create a Databricks managed service principal:

    <iframe title="YouTube video player" />

  * Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html).

    You will need the personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  For [Unstructured Ingest](/open-source/ingestion/overview), the following Databricks authentication types are supported:

  * For Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html): The personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  * For username and password (basic) authentication ([AWS](https://docs.databricks.com/archive/dev-tools/basic.html) only): The user's name and password values.

  * For OAuth machine-to-machine (M2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html)): The client ID and OAuth secret values for the corresponding service principal.

  * For OAuth user-to-machine (U2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-u2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-u2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-u2m.html)): No additional values.

  * For Azure managed identities (formerly Managed Service Identities (MSI) authentication) ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-mi) only): The client ID value for the corresponding managed identity.

  * For Microsoft Entra ID service principal authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-sp) only): The tenant ID, client ID, and client secret values for the corresponding service principal.

  * For Azure CLI authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-cli) only): No additional values.

  * For Microsoft Entra ID user authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/user-aad-token) only): The Entra ID token for the corresponding Entra ID user.

  * For Google Cloud Platform credentials authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-creds.html) only): The local path to the corresponding Google Cloud service account's credentials file.

  * For Google Cloud Platform ID authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-id.html) only): The Google Cloud service account's email address.

* The name of the parent catalog in Unity Catalog for
  [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
  [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
  [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html) for the volume.

* The name of the parent schema (formerly known as a database) in Unity Catalog for
  [AWS](https://docs.databricks.com/schemas/create-schema.html),
  [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
  [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) for the volume.

* The name of the volume in Unity Catalog for [AWS](https://docs.databricks.com/tables/managed.html),
  [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
  [GCP](https://docs.gcp.databricks.com/tables/managed.html), and optionally any path in that volume that you want to access directly, beginning with the volume's root.

* The Databricks workspace user or service principal must have the following *minimum* set of privileges to read from or write to the
  existing volume in Unity Catalog:

  * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
  * `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
  * `READ VOLUME` and `WRITE VOLUME` on the volume.

  The following videos show how to create and set privileges for a catalog, schema (formerly known as a database), and volume in Unity Catalog.

  <iframe title="YouTube video player" />

  Learn more about how to check and set Unity Catalog privileges for
  [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

## Examples

To create a Databricks Volumes destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="databricks_volumes",
                  config={
                      "host": "<host>",
                      "catalog": "<catalog>",
                      "schema": "<schema>",
                      "volume": "<volume>",
                      "volume_path": "<volume_path>",

                      # For Databricks OAuth machine-to-machine (M2M) authentication:
                      "client_secret": "<client-secret>",
                      "client_id": "<client-id>"

                      # For Databricks personal access token authentication:
                      "token": "<token>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "databricks_volumes",
      "config": {
          "host": "<host>",
          "catalog": "<catalog>",
          "schema": "<schema>",
          "volume": "<volume>",
          "volume_path": "<volume_path>",

          # For Databricks OAuth machine-to-machine (M2M) authentication:
          "client_secret": "<client-secret>",
          "client_id": "<client-id>"

          # For Databricks personal access token authentication:
          "token": "<token>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Databricks workspace host URL.

  <Note>
    Do not add a trailing slash (`/`) to the workspace host URL.
  </Note>
</ParamField>

<ParamField type="string">
  For Databricks OAuth machine-to-machine (M2M) authentication, the **Client ID** (or **UUID** or **Application ID**) value for the Databricks managed service principal that has the appropriate privileges to the volume.
</ParamField>

<ParamField type="string">
  For Databricks OAuth M2M authentication, the associated OAuth **Secret** value for the Databricks managed service principal that has the appropriate privileges to the volume.
</ParamField>

<ParamField type="string">
  For Databricks personal access token authentication, the personal access token's value.
</ParamField>

<ParamField type="string">
  The name of the catalog to use.
</ParamField>

<ParamField type="string">
  The name of the associated schema.
</ParamField>

<ParamField type="string">
  The name of the associated volume.
</ParamField>

<ParamField type="string">
  Any optional path to access within the volume.
</ParamField>


# Delta Tables in Amazon S3
Source: https://docs.unstructured.io/api-reference/workflow/destinations/delta-table



<Tip>
  This article covers connecting Unstructured to Delta Tables in Amazon S3. For information about
  connecting Unstructured to Delta Tables in Databricks instead, see
  [Delta Tables in Databricks](/api-reference/workflow/destinations/databricks-delta-table).
</Tip>

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to a Delta Table, stored in Amazon S3.

## Requirements

You will need:

The following video shows how to fulfill the minimum set of Amazon S3 requirements to store Delta Tables:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account.

For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* For authenticated bucket write access or both, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. To grant permissions,
  attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
  To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Provide an AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, provide the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Delta table output format

A Delta table consists of Parquet files that contain data and a transaction log that stores metadata about the transactions.
[Learn more](https://delta-io.github.io/delta-rs/how-delta-lake-works/architecture-of-delta-table/).

The Delta Tables in Amazon S3 destination connector generates the following output within the specified path to the S3 bucket (or the specified folder within the bucket):

* Initially, one Parquet (`.parquet`) file per file in the source location. For example, for a file in the source location named `my-file.pdf`, an associated
  file with the extension `.parquet` is generated. Various kinds of file transactions can result in additional Parquet files being generated. These Parquet filenames are automatically generated by the Delta Lake engine and are not meant to be manually modified.
* A folder named `_delta_log` that contains metadata and change history about the `.parquet` files. As Parquet files are added to, changed, or removed from
  the specified bucket or folder path, the `_delta_log` folder is updated with any related metadata and change history details.

Together, this set of Parquet files and their associated `_delta_log` folder (and its contents) describe a single, versioned Delta table. Because of this, Unstructured recommends the following usage best practices:

* In the source location, each set of source files that is to be considered as a unit for change management purposes should be controlled by a unique, dedicated
  Delta Tables in S3 destination connector. This connector should reference a unique, dedicated output folder within the bucket. Having
  multiple workflows refer to different sets of source files, yet all share the same Delta table, could results in data loss or table corruption.
* Avoid directly modifying, adding, or deleting Parquet data files or the `_delta_log` folder within a Delta table's directory. This can lead to data loss or table corruption.
* If you need to copy or move a Delta table to a different location,
  you must move or copy its entire set of Parquet files and its associated `_delta_log` folder (and its contents) together as a unit.
  Note that the copied or moved Delta table will
  no longer be controlled by the original Delta Tables in S3 destination connector.

## Examples

To create a Delta Tables in Amazon S3 destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="delta_table",
                  config={
                      "aws_region": "<aws-region>",
                      "table_uri": "<table-uri>",
                      "aws_access_key_id": "<aws-access-key-id>",
                      "aws_secret_access_key": "<aws-secret-access-key>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "type": "delta_table",
      "name": "<name>",
      "config": {
          "aws_region": "<aws-region>",
          "table_uri": "<table-uri>",
          "aws_access_key_id": "<aws-access-key-id>",
          "aws_secret_access_key": "<aws-secret-access-key>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

* `<name>` (*required*) - A unique name for this connector.
* `<aws-region>` (*required*) - The AWS Region identifier (for example, `us-east-1`) for the Amazon S3 bucket you want to store the Delta Table in.
* `<table-uri>` (*required*) - The URI of the Amazon S3 bucket you want to store the Delta Table in. This typically takes the format `s3://my-bucket/my-folder`.
* `<aws-access-key-id>` (*required*) - The AWS access key ID for the AWS IAM principal (such as an IAM user) that has the appropriate access to the S3 bucket.
* `<aws-secret-access-key>` (*required*) - The AWS secret access key for the corresponding AWS access key ID.


# Elasticsearch
Source: https://docs.unstructured.io/api-reference/workflow/destinations/elasticsearch



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Elasticsearch.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Elastic Cloud instances are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

* For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

  <iframe title="YouTube video player" />

* For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

  <iframe title="YouTube video player" />

* You will need the name of the index on the instance.

  For the destination connector, if you need to create an index, you can use for example the following `curl` command. Replace the following placeholders:

  * Replace `<host>:<port>` with the instance's host identifier and port number.

  * Replace one of the following, and remove the other:

    * For an instance's user and password combination, replace `<username>` with your Elasticsearch or Elastic Cloud instance's user name, replace `<password>` with your password, and then remove `--user "Authorization: ApiKey <your-api-key>"`.
    * For an Elastic Cloud API key, replace `<your-api-key>` with your Elastic Cloud API key, and then remove `--user "<username>:<password>"`.

  * Replace `<index-name>` with the name of the new index on the instance.

  * Replace `<index-schema>` with the schema for the index. A schema is optional; see the explanation
    following this `curl` command for more information.

  ```bash theme={null}
  curl --request PUT "<host>:<port>/<index-name>" \
  --user "<username>:<password>" \
  --user "Authorization: ApiKey <your-api-key>" \
  [--header "Content-Type: application/json" \
  --data '<index-schema>']
  ```

  To learn more, see [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

  For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
  Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
  to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  For objects in the `metadata` field that Unstructured produces and that you want to store in an Elasticsearch or Elastic Cloud index, you must create fields in your index's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    },
    "entities": {
      "items": [
        {
          "entity": "HOME FOR FUTURE",
          "type": "ORGANIZATION"
        },
        {
          "entity": "221 Queen Street, Melbourne VIC 3000",
          "type": "LOCATION"
        }
      ],
      "relationships": [
        {
          "from": "HOME FOR FUTURE",
          "relationship": "based_in",
          "to": "221 Queen Street, Melbourne VIC 3000"
        }
      ]
    }
  }
  ```

  You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
  required by Unstructured whenever you create your own index schema:

  * `element_id`
  * `record_id`, which is required by Unstructured for intelligent record updates.
  * `type`, which is not required, but highly recommended.
  * `text`
  * `embeddings` if embeddings are generated; make sure to set `dims` to the same number of dimensions as the embedding model generates.

  ```json theme={null}
  {
    "mappings": {
      "properties": {
        "element_id": {
          "type": "keyword"
        },
        "record_id": {
          "type": "text"
        },
        "text": {
          "type": "text"
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "embeddings": {
          "type": "dense_vector",
          "dims": 1536,
          "index": true,
          "similarity": "cosine"
        },
        "metadata": {
          "properties": {
            "is_extracted": {
              "type": "boolean"
            },
            "coordinates-points": {
              "type": "float"
            },
            "coordinates-system": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates-layout_width": {
              "type": "long"
            },
            "coordinates-layout_height": {
              "type": "long"
            },
            "filetype": {
              "type": "keyword"
            },
            "languages": {
              "type": "keyword"
            },
            "page_number": {
              "type": "integer"
            },
            "image_mime_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "data_source-url": {
              "type": "keyword"
            },
            "data_source-record_locator-protocol": {
              "type": "keyword"
            },
            "data_source-record_locator-remote_file_path": {
              "type": "keyword"
            },
            "entities-items": {
              "properties": {
                "entity": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "entities-relationships": {
              "properties": {
                "from": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "relationship": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "to": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
  * [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
  * [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  <iframe title="YouTube video player" />

* For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console or [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

  <Note>
    When generating an Elastic Cloud service instance's API key through the Elasticsearch Service web console, be sure after you click **Create API key** to select **Encoded** to get the API key's value.
    (The **Encoded** API key value is the Base64 encoding of the UTF-8 representation of the API key's ID and key value joined by a colon \[:].)
  </Note>

* For self-managed Elasticsearch, you will need:

  * The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).

  * If you're using basic authentication to the self-managed instance, the user's name and password.

  * If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

    <Note>
      When generating an API key for a self-managed Elasticsearch instance by calling [POST /\_security/api\_key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key), be sure to use
      the `encoded` value in the response payload. (This `encoded` value is the Base64 encoding of the UTF-8 representation of the payload's `id` and `api_key` values joined by a colon \[:]). Do not use the `api_key` value by itself.
    </Note>

  * If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

## Examples

To create an Elasticsearch destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="elasticsearch",
                  config={
                      "hosts": ["<host-url>"],
                      "es_api_key": "<es-api-key>",
                      "index_name": "<index-name>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "elasticsearch",
      "config": {
          "hosts": ["<host-url>"],
          "es_api_key": "<es-api-key>",
          "index_name": "<index-name>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  An array of Elastic Cloud instance URLs, for example `["https://<random-number>.<region>.<cloud-provider>.cloud.es.io"]`.
</ParamField>

<ParamField type="string">
  The Elastic Cloud API key for the target cluster.
</ParamField>

<ParamField type="string">
  The name of the target index in the cluster.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Finding Needles in a Haystack: PII Detection at Scale with Unstructured, Box, and Elasticsearch](https://unstructured.io/blog/finding-needles-in-a-haystack-pii-detection-at-scale-with-unstructured-box-and-elasticsearch)


# IBM FileNet
Source: https://docs.unstructured.io/api-reference/workflow/destinations/filenet

Connect Unstructured to IBM FileNet as a destination to send processed document data into your FileNet content repository.

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* An [IBM Cloud Pak for Business Automation as a Service](https://www.ibm.com/products/cloud-pak-for-business-automation) account with access to the IBM FileNet [Content Platform Engine server](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=architecture-content-platform-engine) object store to which you want to connect. IBM FileNet is a component of IBM Cloud Pak for Business Automation as a Service.

To access the information you'll need to configure the connector:

1. Log into your [account](https://www.automationcloud.ibm.com/auth/index.jsp).

2. Choose the **Navigator** tile.

   The [IBM Navigator](https://www.ibm.com/docs/en/content-navigator/3.2.0) launches in a separate browser window, and provides a view of your object stores and content. You can use the IBM Navigator views to find the information necessary to create a connection to Unstructured.

   For the URL of your IBM FileNet server:

   * The server URL displays in the browser address bar. You only need the base URL that specifies the company and domain. For example, `https://<company-name>.automationcloud.ibm.com`.

   For [object store](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=infrastructure-defining-object-stores) names and folder paths:

   * Select the folder in the left pane. The full folder path is displayed at the top of the main detail pane, in the following format: `<object-store>/<folder>/etc`.

   For the document class:

   * Right-click the document and select **Properties**.

   For the account username:

   * Right-click the profile icon on the upper right in the top menu.

## Examples

To create an IBM FileNet destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="filenet",
                  config={
                       "server_url": "<server-url>",
                       "object_store": "<object-store>",
                       "target_folder": "<target_folder>",
                       "document_class": "<document-class>",
                       "username": "<username>",
                       "password": "<password>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  # ...
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "filenet",
      "config": {
          "server_url": "<server-url>",
          "object_store": "<object-store>",
          "target_folder": "<target-folder>",
          "document_class": "<document-class>",
          "username": "<username>",
          "password": "<password>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The base URL of your Content Platform Engine, containing both the IBM domain and your company's subdomain. For example, `https://<company-name>.automationcloud.ibm.com`.
</ParamField>

<ParamField type="string">
  The name of the object store to connect within the Content Platform Engine.
</ParamField>

<ParamField type="string">
  Source connector only. The path of the folder within the object store to use as the source.
</ParamField>

<ParamField type="string">
  Destination connector only. The path of the folder within the object store to use as the upload destination.
</ParamField>

<ParamField type="string">
  The class of documents to include.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to include documents contained in any subfolders.
</ParamField>

<ParamField type="string">
  The username of the IBM Cloud Pak for Business Automation as a Service account to use.
</ParamField>

<ParamField type="string">
  The password for the corresponding username.
</ParamField>


# Google Cloud Storage
Source: https://docs.unstructured.io/api-reference/workflow/destinations/google-cloud



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Google Cloud Storage.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

* A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
  [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:
    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```
  * For Windows:
    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

  This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

  * `Storage Object Viewer` for bucket read access.
  * `Storage Object User` for bucket write access.
  * The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

  To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
  in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

## Examples

To create a Google Cloud Storage destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="gcs",
                  config={
                      "remote_url": "<remote-url>",
                      "service_account_key": "<service-account-key>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "type": "gcs",
      "name": "<name>",
      "config": {
          "remote_url": "<remote-url>",
          "service_account_key": "<service-account-key>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The contents of a service account key file, expressed as a single string without line breaks, for a Google Cloud service account that has the required access permissions to the bucket.
</ParamField>

<ParamField type="string">
  The URI for the Google Cloud Storage bucket and any target folder path within the bucket. This URI takes the format `gs://<bucket-name>[/folder-name]`.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to ingest data recursively from any subfolders, starting from the path specified by `remote_url`.
</ParamField>


# IBM watsonx.data
Source: https://docs.unstructured.io/api-reference/workflow/destinations/ibm-watsonxdata



<Tip>
  The IBM watsonx.data destination connector relies on an Apache Iceberg-based catalog within the watsonx.data data store instance.
  Apache Iceberg is suitable for managed data storage and cataloging, but not for embedding storage or semantic similarity
  queries. For embedding storage and semantic similarity queries, Unstructured recommends that you use the following destination connectors
  instead:

  * [Astra DB](/api-reference/workflow/destinations/astradb)
  * [Milvus](/api-reference/workflow/destinations/milvus) on IBM watsonx.data
</Tip>

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to IBM watsonx.data.

## Requirements

You will need:

* An [IBM Cloud account](https://cloud.ibm.com/login). [Create an IBM Cloud account](https://cloud.ibm.com/registration) if you do not already have one.

* An API key for the IBM Cloud account. If you do not have one already, create one as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. In the top navigation bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
  3. On the sidebar, under **Manage identities**, click **API keys**.
  4. With the **View** list showing **My IBM Cloud API keys**, click **Create**.
  5. Enter some **Name** and an optional **Description** for the API key.
  6. Leave **Leaked action** set to **Disable the leaked key** and **Session management** set to **No**.
  7. Click **Create**.
  8. Click **Copy** or **Download** to copy or save the API key to a secure location. You won't be able to access this API key from this screen again.

* An IBM Cloud Object Storage (COS) instance in the account, and a bucket within that instance. If you do not have them already,
  create them as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. Click **Create resource**.
  4. With **IBM Cloud catalog** selected, search for and select **Object Storage**.
  5. Complete the on-screen instructions to finish creating the COS instance.
  6. With the COS instance's settings page shown, on the **Buckets** tab, click **Create bucket**.
  7. Complete the on-screen instructions to finish creating the bucket.

* The name, region, and public endpoint for the target bucket within the target Cloud Object Storage (COS) instance. To get these:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Storage**, and then click the target COS instance.
  4. On the **Buckets** tab, click the target bucket.
  5. On the **Configuration** tab, note the following:

     * Under **Bucket details**, note the **Bucket name**. This is the bucket's name.
     * Under **Bucket details** section, note the value inside of the parentheses inside **Location**, for example `us-east`. This is the bucket's region.
     * Under **Endpoints**, note the value of **Public**, for example `s3.us-east.cloud-object-storage.appdomain.cloud`. (Ignore the values of
       **Private** and **Direct**). This is the bucket's public endpoint.

* An HMAC access key ID and secret access key for the target Cloud Object Storage (COS) instance. If you do not have them already,
  get or create them as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.

  3. In the list of resources, expand **Storage**, and then click the target COS instance.

  4. On the **Service credentials** tab, if there is a credential that you want to use in the list, expand the credential, and copy the following values to a secure location:

     * `access_key_id` under `cos_hmac_keys`, which represents the HMAC access key ID.
     * `secret_access_key` under `cos_hmac_keys`, which represents the HMAC secret access key.

     After you have copied the preceding values, you have completed this procedure.

  5. If there is not a credential that you want to use, or there are no credentials at all, click **New Credential**.

  6. Enter some **Name** for the credential.

  7. For **Role**, select at least **Writer**, leave **Select Service ID** set to **Auto Generated**,
     switch on **Include HMAC Credential**, and then click **Add**.

  8. In the list of credentials, expand the credential, and copy the following values to a secure location:

     * `access_key_id` under `cos_hmac_keys`, which represents the HMAC access key ID.
     * `secret_access_key` under `cos_hmac_keys`, which represents the HMAC secret access key.

* An IBM watsonx.data data store instance in the IBM Cloud account. If you do not have one already, create one as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. Click **Create resource**.
  4. With **IBM Cloud catalog** selected, search for and select **watsonx.data**.
  5. Complete the on-screen instructions to finish creating the watsonx.data data store instance.

* An Apache Iceberg-based catalog within the watsonx.data data store instance. If you do not have one already, create one as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.

  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.

  4. Click **Open web console**.

  5. If prompted, log in to the web console.

  6. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.

  7. Click **Add component**.

  8. Under **Storage**, click **IBM Cloud Object Storage**, and then click **Next**.

  9. Complete the on-screen instructions to finish creating the Iceberg catalog. This includes providing the following settings:

     If you select **Discover COS instance**, you must provide the following settings:

     * The name of the target COS instance.
     * The name of the target storage object (such as the target bucket) within the COS instance.
     * Some display name for the component.
     * After you provide this information, do the following:

       a. Check the box labelled **Associate Catalog**.<br />
       b. For **Catalog type**, select **Apache Iceberg**.<br />
       c. Enter some **Catalog name**.<br />
       d. Click **Associate**.<br />

     If you select **Register my own**, you must provide the following settings:

     * Some display name for the component.
     * The name of the target bucket within the target Cloud Object Storage (COS) instance that you noted earlier.
     * The region for the target bucket, which you noted earlier.
     * The public endpoint for the target bucket, which you noted earlier. For this screen only, be sure to prefix the public endpoint with `https://`.
     * The HMAC access key ID for the target COS instance, which you noted earlier.
     * The HMAC secret access key for the target COS instance, which you noted earlier.
     * After you provide this information, do the following:

       a. Check the box labelled **Associate Catalog**.<br />
       b. For **Catalog type**, select **Apache Iceberg**.<br />
       c. Enter some **Catalog name**.<br />
       d. Click **Associate**.<br />

  10. On the sidebar, click **Infrastructure manager**. Make sure the catalog is associated with the appropriate engines. If it is not, rest your mouse
      on an unassociated target engine, click the **Manage associations** icon, check the box next to the target catalog's name, and then
      click **Save and restart engine**.

      To create an engine if one is not already shown, click **Add component**, and follow the on-screen to add an appropriate engine from the list of available **Engines**
      (for example, an **IBM Presto** engine).

* The catalog name and metastore REST endpoint for the target Iceberg catalog. To get this:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. In the **Catalogs** section, click the target Iceberg catalog.
  8. On the **Details** tab, note the value of **Name** representing the catalog name, and **Metastore REST endpoint** representing the metastore REST endpoint. (Ignore the **Metastore Thrift endpoint** value.)

* A namespace (also known as a schema) and a table in the target catalog. If you do not have these already, create them as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Data manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. On the **Browse data** tab, under **Catalogs associated**, click the target catalog.
  8. Click the ellipses, and then click **Create schema**.
  9. Enter some **Name** for the schema, and then click **Create**.
  10. On the sidebar, click **Query workspace**.
  11. In the SQL editor, enter and run a table creation statement such as the following one that uses
      [Presto SQL](https://prestodb.io/docs/current/connector/iceberg.html) syntax, replacing `<catalog-name>` with the name of the target
      catalog and `<schema-name>` with the name of the target schema, with at minimum the fields `id`, `type`, `text`, `element_id`, and `record_id`:

      ```sql theme={null}
      CREATE TABLE <catalog-name>.<schema-name>.elements (
         id varchar,
         type varchar,
         text varchar,
         element_id varchar,
         record_id varchar,
         <additional-fields-as-needed>
      )
      WITH (
         delete_mode = 'copy-on-write',
         format = 'PARQUET',
         format_version = '2'
      )
      ```

      <Info>
        The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
      </Info>

      For objects in the `metadata` field that Unstructured produces and that you want to store in an Apache Iceberg table in IBM watsonx.data,
      you must create fields in your table schema that follows Unstructured's `metadata` field naming convention. For example, if Unstructured
      produces a `metadata` field with the following child objects:

      ```json theme={null}
      "metadata": {
        "is_extracted": "true",
        "coordinates": {
          "points": [
            [
              134.20055555555555,
              241.36027777777795
            ],
            [
              134.20055555555555,
              420.0269444444447
            ],
            [
              529.7005555555555,
              420.0269444444447
            ],
            [
              529.7005555555555,
              241.36027777777795
            ]
          ],
          "system": "PixelSpace",
          "layout_width": 1654,
          "layout_height": 2339
        },
        "filetype": "application/pdf",
        "languages": [
          "eng"
        ],
        "page_number": 1,
        "image_base64": "aaa...bbb...111",
        "image_mime_type": "image/jpeg",
        "filename": "realestate.pdf",
        "data_source": {
          "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
          "record_locator": {
            "protocol": "file",
            "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
          }
        },
        "entities": {
          "items": [
            {
              "entity": "HOME FOR FUTURE",
              "type": "ORGANIZATION"
            },
            {
              "entity": "221 Queen Street, Melbourne VIC 3000",
              "type": "LOCATION"
            }
          ],
          "relationships": [
            {
              "from": "HOME FOR FUTURE",
              "relationship": "based_in",
              "to": "221 Queen Street, Melbourne VIC 3000"
            }
          ]
        }
      }
      ```

      You could create corresponding fields in your table schema with the following field names and data types:

      | **Field name**    | **Apache Iceberg data type**                                                                                                        |
      | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
      | `is_extracted`    | `varchar`                                                                                                                           |
      | `points`          | `varchar`                                                                                                                           |
      | `system`          | `varchar`                                                                                                                           |
      | `layout_width`    | `bigint`                                                                                                                            |
      | `layout_height`   | `bigint`                                                                                                                            |
      | `filetype`        | `varchar`                                                                                                                           |
      | `languages`       | `array(varchar)`                                                                                                                    |
      | `page_number`     | `varchar`                                                                                                                           |
      | `image_base64`    | `varchar`                                                                                                                           |
      | `image_mime_type` | `varchar`                                                                                                                           |
      | `filename`        | `varchar`                                                                                                                           |
      | `url`             | `varchar`                                                                                                                           |
      | `record_locator`  | `varchar`                                                                                                                           |
      | `entities`        | `row(items array(row(entity varchar, type varchar)), relationships array(row("from" varchar, relationship varchar, "to" varchar)))` |

      Incoming elements that do not have matching column
      names will be dropped upon record insertion. For example, if the incoming data has an element named `sent_from` and there is no
      column named `sent_from` in the table, the `sent_from` element will be dropped upon record insertion. You should modify the preceding
      sample table creation statement to add columns for any additional elements that you want to be included upon record
      insertion.

      To increase query performance, Iceberg uses [hidden partitioning](https://iceberg.apache.org/docs/latest/partitioning/) to
      group similar rows together when writing. You can also
      [explicitly define partitions](https://prestodb.io/docs/current/connector/iceberg.html#create-table) as part of the
      preceding `CREATE TABLE` statement.

* The name of the target namespace (also known as a schema) within the target catalog, and name of the target table within that schema. To get these:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Data manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. On the **Browse data** tab, expand the name of the target catalog, and note the names of the target schema and target table.

* The name of the column in the target table that uniquely identifies each of the records in the table.

* To improve performance, the target table should be set to regularly remove old metadata files. To do this, run the following Python script.
  (You cannot use the preceding `CREATE TABLE` statement, or other SQL statements such as `ALTER TABLE`, to set this behavior.) To get the
  values for the specified environment variables, see the preceding instructions.

  ```python theme={null}
  # Improves performance by setting the target table to regularly remove 
  # old metadata files. 
  #
  # First, install the following dependencies into your Python virtual 
  # environment:
  # 
  # pip install requests pyiceberg pyarrow
  #
  # Then, set the following environment variables:
  #
  # IBM_IAM_API_KEY - An API key value for the target IBM Cloud account.
  # IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT - The metastore REST endpoint 
  #     value for the target Apache Iceberg catalog in the target IBM watsonx.data 
  #     data store instance.
  # IBM_COS_BUCKET_PUBLIC_ENDPOINT - The target IBM Cloud Object Storage (COS) 
  #     instance’s endpoint value.
  # IBM_COS_ACCESS_KEY - An HMAC access key ID for the target COS instance.
  # IBM_COS_SECRET_ACCESS_KEY - The associated HMAC secret access key ID for the 
  #     target HMAC access key.
  # IBM_COS_BUCKET_REGION - The target COS instance’s region short ID.
  # IBM_ICEBERG_CATALOG - The name of the target Iceberg catalog.
  # IBM_ICEBERG_SCHEMA - The name of the target namespace (also known as a schema) 
  #     in the target catalog.
  # IBM_ICEBERG_TABLE - The name of the target table in the target schema.
  #
  # To get these values, see the Unstructured documentation for the 
  #     IBM watsonx.data connector.

  import os
  import requests
  from pyiceberg.catalog import load_catalog

  def main():
     # Get a bearer token for the target IBM Cloud account.   
     bearer_token = requests.post(
        url="https://iam.cloud.ibm.com/identity/token",
        headers={
              "Content-Type": "application/x-www-form-urlencoded",
              "Accept": "application/json"
        },
        data={
              "grant_type": "urn:ibm:params:oauth:grant-type:apikey", 
              "apikey": os.getenv("IBM_IAM_API_KEY")
        }
     ).json().get("access_token")

     # Connect to the target Iceberg catalog.
     catalog = load_catalog(
        os.getenv("IBM_ICEBERG_CATALOG"),
        **{
              "type": "rest",
              "uri": f"https://{os.getenv("IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT")}/mds/iceberg",
              "token": bearer_token,
              "warehouse": os.getenv("IBM_ICEBERG_CATALOG"),
              "s3.endpoint": os.getenv("IBM_COS_BUCKET_PUBLIC_ENDPOINT"),
              "s3.access-key-id": os.getenv("IBM_COS_ACCESS_KEY"),
              "s3.secret-access-key": os.getenv("IBM_COS_SECRET_ACCESS_KEY"),
              "s3.region": os.getenv("IBM_COS_BUCKET_REGION")
        },
     )
              
     # Load the target table.
     table = catalog.load_table(f"{os.getenv("IBM_ICEBERG_SCHEMA")}.{os.getenv("IBM_ICEBERG_TABLE")}")

     # Set the target table's properties to remove old metadata files.
     with table.transaction() as transaction:
        transaction.set_properties(
              {
                 "commit.manifest.min-count-to-merge": 10,
                 "commit.manifest-merge.enabled": True,
                 "write.metadata.previous-versions-max": 10,
                 "write.metadata.delete-after-commit.enabled": True,
              }
        )

     # Confirm that the target table's properties were set as expected.
     print(table.metadata.properties)

  if __name__ == "__main__":
     main()
  ```

## Examples

To create an IBM watsonx.data destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="ibm_watsonx_s3",
                  config={
                      "iceberg_endpoint": "<iceberg-endpoint>",
                      "object_storage_endpoint": "<object-storage-endpoint>",
                      "object_storage_region": "<object-storage-region>",
                      "iam_api_key": "<iam-api-key>",
                      "access_key_id": "<access-key-id>",
                      "secret_access_key": "<secret-access-key>",
                      "catalog": "<catalog>",
                      "namespace": "<namespace>",
                      "table": "<table>",
                      "max_retries": <max-retries>,
                      "max_retries_connection": <max-retries-connection>,
                      "record_id_key": "<record-id-key>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "ibm_watsonx_s3",
      "config": {
          "iceberg_endpoint": "<iceberg-endpoint>",
          "object_storage_endpoint": "<object-storage-endpoint>",
          "object_storage_region": "<object-storage-region>",
          "iam_api_key": "<iam-api-key>",
          "access_key_id": "<access-key-id>",
          "secret_access_key": "<secret-access-key>",
          "catalog": "<catalog>",
          "namespace": "<namespace>",
          "table": "<table>",
          "max_retries": <max-retries>,
          "max_retries_connection": <max-retries-connection>,
          "record_id_key": "<record-id-key>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The metastore REST endpoint for the target Apache Iceberg-based catalog within the IBM watsonx.data data store instance. Do not include `https://` in this value.
</ParamField>

<ParamField type="string">
  The public endpoint for the target bucket within the IBM Cloud Object Storage (COS) instance that is associated with the catalog. Do not include `https://` in this value.
</ParamField>

<ParamField type="string">
  The region short ID (such as us-east) for the bucket.
</ParamField>

<ParamField type="string">
  A valid API key value for the IBM Cloud account.
</ParamField>

<ParamField type="string">
  A valid hash-based message authentication code (HMAC) access key ID for the COS instance.
</ParamField>

<ParamField type="string">
  The HMAC secret access key for the access key ID.
</ParamField>

<ParamField type="string">
  The name of the target Apache Iceberg-based catalog within the IBM watsonx.data data store instance.
</ParamField>

<ParamField type="string">
  The name of the target namespace (also known as a schema) within the catalog.
</ParamField>

<ParamField type="string">
  The name of the target table within the namespace (schema).
</ParamField>

<ParamField type="integer">
  The maximum number of retries for the upload process. Typically, an optimal setting is `150`. If specified, it must be a number between `2` and `500`, inclusive.
</ParamField>

<ParamField type="integer">
  The maximum number of retries when connecting to the catalog. Typically, an optimal setting is `15`. If specified, it must be a number between `2` and `100`, inclusive.
</ParamField>

<ParamField type="string">
  The name of the column that uniquely identifies each record in the target table.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Unstructured + IBM watsonx.data: A New OEM Partnership Powering the Future of Enterprise AI](https://unstructured.io/blog/unstructured-ibm-watsonx-data-a-new-oem-partnership-powering-the-future-of-enterprise-ai)


# Kafka
Source: https://docs.unstructured.io/api-reference/workflow/destinations/kafka



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Kafka.

## Requirements

You will need:

* A Kafka cluster in [Confluent Cloud](https://www.confluent.io/confluent-cloud).
  ([Create a cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#create-ak-clusters).)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

  <iframe title="YouTube video player" />

* The [hostname and port number](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#view-a-ak-cluster) of the bootstrap Kafka cluster to connect to..

* The name of the topic to read messages from or write messages to on the cluster.
  [Create a topic](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).
  [Access available topics](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).

* For authentication, an [API key and secret](https://docs.confluent.io/cloud/current/security/authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.html#add-an-api-key).

## Examples

To create a Kafka destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="kafka-cloud",
                  config={
                      "bootstrap_servers": "<bootstrap-server>",
                      "port": <port>,
                      "group_id": "<group-id>",
                      "kafka_api_key": "<kafka-api-key>",
                      "secret": "<secret>",
                      "topic": "<topic>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "kafka-cloud",
      "config": {
          "bootstrap_server": "<bootstrap-server>",
          "port": <port>,
          "group_id": "<group-id>",
          "kafka_api_key": "<kafka-api-key>",
          "secret": "<secret>",
          "topic": "<topic>",
          "num_messages_to_consume": <num-messages-to-consume>
      },
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The hostname of the bootstrap Kafka cluster to connect to.
</ParamField>

<ParamField type="integer">
  The port number of the bootstrap Kafka cluster to connect to.
</ParamField>

<ParamField type="string">
  The ID of the consumer group. A consumer group is a way to allow a pool of consumers to divide the consumption of data over topics and partitions.
</ParamField>

<ParamField type="string">
  For authentication, the API key for access to the cluster.
</ParamField>

<ParamField type="string">
  For authentication, the secret for access to the cluster.
</ParamField>

<ParamField type="string">
  The name of the topic to read messages from or write messages to on the cluster.
</ParamField>

<ParamField type="integer">
  Destination connector only. The maximum number of messages to send in a single batch.
</ParamField>

<ParamField type="integer">
  Source connector only. The maximum number of messages that the consumer will try to consume.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Unstructured Platform Now Integrates with Apache Kafka in Confluent Cloud](https://unstructured.io/blog/unstructured-platform-now-integrates-with-apache-kafka-in-confluent-cloud)


# Local
Source: https://docs.unstructured.io/api-reference/workflow/destinations/local



Send processed data from Unstructured to local storage.

This local destination connector is supported only for REST API clients such as `curl` and Postman. It is currently not supported by Unstructured Pipelines or the Unstructured Python SDK.

The local destination connector works only with the [local source connector](/api-reference/workflow/sources/local). You cannot use the local destination connector with any other source connector.

For details, see the following:

1. [Create a workflow](/api-reference/workflow/workflows#create-a-workflow) that uses a local destination and local source.
2. [Run a workflow as a job](/api-reference/api/workflow/run-workflow) that uses a local destination and local source.
3. [Download a processed local file from the workflow's completed job run](/api-reference/api/job/download-job-output).


# Milvus
Source: https://docs.unstructured.io/api-reference/workflow/destinations/milvus



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Milvus.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Milvus cloud-based instances (such as Milvus on IBM watsonx.data, or Zilliz Cloud) are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Milvus local and cloud-based instances are supported.

* For Milvus on IBM watsonx.data, you will need:

  <iframe title="YouTube video player" />

  * An [IBM Cloud account](https://cloud.ibm.com/registration).

  * An IBM watsonx.data [Lite plan](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_prov_lite_1)
    or [Enterprise plan](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started_1) within your IBM Cloud account.

    * If you are provisioning a Lite plan, be sure to choose the **Generative AI** use case when prompted, as this is the only use case offered that includes Milvus.

  * A [Milvus service instance in IBM watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-adding-milvus-service).

    * If you are creating a Milvus service instance within a watsonx.data Lite plan, when you are prompted to choose a Milvus instance size, you can only select **Lite**. Because the Lite
      Milvus instance size is recommended only for 384 dimensions, you should also use an embedding model that uses 384 dimensions only.
    * If you are creating a Milvus service instance within a watsonx.data Enterprise plan, you can choose any available Milvus instance size. However, all Milvus instance sizes other than
      **Custom** are recommended only for 384 dimensions, which means you should use an embedding model that uses 384 dimensions only.
      The **Custom** Milvus instance size is recommended for any number of dimensions.

  * The URI of the instance, which takes the format of `https://`, followed by the instance's **GRPC host**, followed by a colon and the **GRPC port**.
    This takes the format of `https://<host>:<port>`. To get this information, do the following:

    a. Sign in to your IBM Cloud account.<br />
    b. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the title bar.<br />
    c. Expand **Databases**, and then click the name of the target **watsonx.data** plan.<br />
    d. Click **Open web console**.<br />
    e. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the title bar.<br />
    f. Click the target Milvus service instance.<br />
    g. On the **Details** tab, under **Type**, click **View connect details**.<br />
    h. Under **Service details**, expand **GRPC**, and note the value of **GRPC host** and **GRPC port**.<br />

  * The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.

  * The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database. Note the collection requirements at the end of this section.

  * The username and password to access the instance.

    * The username for Milvus on IBM watsonx.data is typically `ibmlhapikey`.

      <Note>
        More recent versions of Milvus on IBM watsonx.data require `ibmlhapikey_<your-IBMid>` instead, where `<your-IBMid>` is
        your IBMid, for example `me@example.com`. To get your IBMid, do the following:

        1. Sign in to your IBM Cloud account.
        2. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
        3. In the sidebar, expand **Manage identities**, and then click **Users**.
        4. In the list of users, click your user name.
        5. On the **User details** tab, in the **Details** tile, note the value of **IBMid**.
      </Note>

    * The password for Milvus on IBM watsonx.data is in the form of an IBM Cloud user API key. To create an IBM Cloud user API key:

      a. Sign in to your IBM Cloud account.<br />
      b. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.<br />
      c. On the sidebar, under **Manage identities**, click **API keys**. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the title bar.<br />
      d. Click **Create**.<br />
      e. Enter some **Name** for the API key.<br />
      f. Optionally, enter some **Description** for the API key.<br />
      g. For **Leaked action**, leave **Disable the leaked key** selected.<br />
      h. For **Session management**, leave **No** selected.<br />
      i. Click **Create**.<br />
      j. Click **Download** (or **Copy**), and then download the API key to a secure location (or paste the copied API key into a secure location). You won't be able to access this API key from this dialog again. If you lose this API key, you can create a new one (and you should then delete the old one).<br />

* For Zilliz Cloud, you will need:

  <iframe title="YouTube video player" />

  * A [Zilliz Cloud account](https://cloud.zilliz.com/signup).

  * A [Zilliz Cloud cluster](https://docs.zilliz.com/docs/create-cluster).

  * The URI of the cluster, also known as the cluster's *public endpoint*, which takes a format such as
    `https://<cluster-id>.<cluster-type>.<cloud-provider>-<region>.cloud.zilliz.com`. To get this public endpoint value, do the following:

    1. After you sign in to your Zilliz Cloud account, on the sidebar, in the list of available projects, select the project that contains the cluster.
    2. On the sidebar, click **Clusters**.
    3. Click the tile for the cluster.
    4. On the **Cluster Details** tab, on the **Connect** subtab, copy the **Public Endpoint** value.

  * The username and password to access the cluster, as follows:

    1. After you sign in to your Zilliz Cloud account, on the sidebar, in the list of available projects, select the project that contains the cluster.
    2. On the sidebar, click **Clusters**.
    3. Click the tile for the cluster.
    4. On the **Users** tab, copy the name of the user.
    5. Next to the user's name, under **Actions**, click the ellipsis (three dots) icon, and then click **Reset Password**.
    6. Enter a new password for the user, and then click **Confirm**. Copy this new password.

  * The name of the [database](https://docs.zilliz.com/docs/database#create-database) in the instance.

  * The name of the [collection](https://docs.zilliz.com/docs/manage-collections-console#create-collection) in the database.

    The collection must have a defined schema before Unstructured can write to the collection. The minimum viable
    schema for Unstructured contains only the fields `element_id`, `embeddings`, `record_id`, and `text`, as follows.
    `type` is an optional field, but highly recommended. For settings for additional Unstructured-produced fields,
    such as the ones within `metadata`, see the usage notes toward the end of this section and adapt them to your specific needs.

    | Field Name                       | Field Type        | Max Length | Dimension |
    | -------------------------------- | ----------------- | ---------- | --------- |
    | `element_id` (primary key field) | **VARCHAR**       | `200`      | --        |
    | `embeddings` (vector field)      | **FLOAT\_VECTOR** | --         | `384`     |
    | `record_id`                      | **VARCHAR**       | `200`      | --        |
    | `text`                           | **VARCHAR**       | `65536`    | --        |
    | `type`                           | **VARCHAR**       | `200`      | --        |

    In the **Create Index** area for the collection, next to **Vector Fields**, click **Edit Index**. Make sure that for the
    `embeddings` field, the **Field Type** is set to **FLOAT\_VECTOR** and the **Metric Type** is set to **Cosine**.

    <Warning>
      The number of dimensions for the `embeddings` field must match the number of dimensions for the embedding model that you plan to use.
    </Warning>

    <Note>
      Fields with a **VARCHAR** data type are limited to a maximum length of 65,535 characters. Attempting to exceed this character count
      will cause Unstructured to throw errors when attempting to write to a Milvus collection, and the associated Unstructured job could fail.
      For example, `metadata` fields that typically exceed these character counts include `image_base64` and `orig_elements`.
    </Note>

    <Info>
      The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
    </Info>

* For Milvus local, you will need:

  * A [Milvus instance](https://milvus.io/docs/install-overview.md).
  * The [URI](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of the instance.
  * The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.
  * The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database.
    Note the collection requirements at the end of this section.
  * The [username and password, or token](https://milvus.io/docs/authenticate.md) to access the instance.

All Milvus instances require the target collection to have a defined schema before Unstructured can write to the collection. The minimum viable
schema for Unstructured contains only the fields `element_id`, `embeddings`, `record_id`, and `text`, as follows.
`type` is an optional field, but highly recommended.

This example code demonstrates the use of the
[Python SDK for Milvus](https://pypi.org/project/pymilvus/) to create a collection with this schema,
targeting Milvus on IBM watsonx.data. For the `MilvusClient` arguments to connect to other types of Milvus deployments, see your Milvus provider's documentation:

```python Python theme={null}
import os

from pymilvus import (
    MilvusClient,
    FieldSchema,
    DataType,
    CollectionSchema
)

DATABASE_NAME   = "default"
COLLECTION_NAME = "my_collection"

client = MilvusClient(
    uri="https://" +
        os.getenv("MILVUS_USER") + 
        ":" + 
        os.getenv("MILVUS_PASSWORD") + 
        "@" + 
        os.getenv("MILVUS_GRPC_HOST") + 
        ":" + 
        os.getenv("MILVUS_GRPC_PORT"),
    db_name=DATABASE_NAME
)

# IMPORTANT: The number of dimensions for the "embeddings" field that
# follows must match the number of dimensions for the embedding model 
# that you plan to use.
fields = [
    FieldSchema(name="element_id", dtype=DataType.VARCHAR, is_primary=True, max_length=200),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=384),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=200),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="type", dtype=DataType.VARCHAR,max_length=200, nullable=True) # Optional, but highly recommended.
]

schema = CollectionSchema(fields=fields)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema,
    using=DATABASE_NAME
)

index_params = client.prepare_index_params()

index_params.add_index(
    field_name="embeddings",
    metric_type="COSINE",
    index_type="IVF_FLAT",
    params={"nlist": 1024}
)

client.create_index(
    collection_name=COLLECTION_NAME,
    index_params=index_params
)

client.load_collection(collection_name=COLLECTION_NAME)
```

For objects in the `metadata` field that Unstructured produces and that you want to store in Milvus, you must create fields in your Milvus collection schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You could create corresponding fields in your Milvus collection schema by using the following field names, data types, and related settings for the
Python SDK for Milvus:

```python theme={null}
# ...
fields = [
    # Define minimum viable required fields: "element_id", "embeddings", "record_id", and "text".
    # "type" is an optional field, but highly recommended.
    FieldSchema(name="element_id", dtype=DataType.VARCHAR, is_primary=True, max_length=200),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=1536),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=200),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="type",dtype=DataType.VARCHAR,max_length=200, nullable=True),
    # Define optional fields, such as these ones related to the "metadata" field.
    FieldSchema(name="is_extracted", dtype=DataType.VARCHAR, max_length=5, nullable=True),
    FieldSchema(name="coordinates_points", dtype=DataType.JSON, nullable=True),
    FieldSchema(name="coordinates_system", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="coordinates_layout_width", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="coordinates_layout_height", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="filetype", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="languages", dtype=DataType.ARRAY, element_type=DataType.VARCHAR, max_length=64, max_capacity=10, nullable=True),
    FieldSchema(name="page_number", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="image_mime_type", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="filename", dtype=DataType.VARCHAR, max_length=256, nullable=True),
    FieldSchema(name="data_source_url", dtype=DataType.VARCHAR, max_length=1024, nullable=True),
    FieldSchema(name="data_source_record_locator_protocol", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="data_source_record_locator_remote_file_path", dtype=DataType.VARCHAR, max_length=1024, nullable=True),
    FieldSchema(name="entities_items", dtype=DataType.JSON, nullable=True),
    FieldSchema(name="entities_relationships", dtype=DataType.JSON, nullable=True)
]
# ...
```

<Note>
  Fields with a `DataType.VARCHAR` data type are limited to a maximum length of 65,535 characters. Attempting to exceed this character count
  will cause Unstructured to throw errors when attempting to write to a Milvus collection, and the associated Unstructured job could fail.
  For example, `metadata` fields that typically exceed these character counts include `image_base64` and `orig_elements`.
</Note>

## Examples

To create a Milvus destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="milvus",
                  config={
                      "user": "<user>",
                      "uri": "<uri>",
                      "db_name": "<db-name>",
                      "password": "<password>",
                      "collection_name": "<collection-name>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "milvus",
      "config": {
          "user": "<user>",
          "uri": "<uri>",
          "db_name": "<db-name>",
          "password": "<password>",
          "collection_name": "<collection-name>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The username to access the Milvus instance.
</ParamField>

<ParamField type="string">
  The URI of the instance, for example: `https://12345.serverless.gcp-us-west1.cloud.zilliz.com.`
</ParamField>

<ParamField type="string">
  The name of the database in the instance.
</ParamField>

<ParamField type="string">
  The password corresponding to the username to access the instance.
</ParamField>

<ParamField type="string">
  The name of the collection in the database.
</ParamField>


# MongoDB
Source: https://docs.unstructured.io/api-reference/workflow/destinations/mongodb



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to MongoDB.

## Requirements

You will need:

<iframe title="YouTube video player" />

The MongoDB requirements for a MongoDB Atlas deployment include:

<Warning>
  For MongoDB Atlas, SCRAM-SHA-1 is not supported for authentication. This means that cluster types that only
  include SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, and **Serverless**, are **not** supported.
  Unstructured only supports SCRAM-SHA-256 for MongoDB Atlas, which is cryptographically stronger than SCRAM-SHA-1.

  If you try to test or use a connector that refers to a cluster type that only includes SCRAM-SHA-1, the
  operation will fail, and you will get an error message similar to the following:
  `[digital envelope routines] unsupported`.
</Warning>

* A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).

* A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). Be sure to **not**
  select a cluster type that only includes SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, or **Serverless**.

* The cluster must be reachable from your application environment, for example by adding IP addresses to your IP access list. [Learn more](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).

* The cluster must be configured to allow IP address. [Learn how](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries).

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

* The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).

* The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).

* The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).

  <Note>
    For the destination connector, Unstructured recommends that all documents in the target collection have a field
    named `record_id` with a `String` data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the collection or, in some cases, the operation could fail altogether.
  </Note>

* The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

  ```text theme={null}
  mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
  ```

  To get the connection string in MongoDB Atlas, do the following:

  1. Log in to your MongoDB Atlas console.
  2. In the sidebar, under **Databases**, click **Clusters**.
  3. Click on the cluster you want to connect to.
  4. Click **Connect**.
  5. Click **Drivers**.
  6. Under **Add your connection string into your application code**, copy the connection string.
     You can then close the **Connect** dialog in MongoDB Atlas.

     Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

  [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

## Examples

To create a MongoDB destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="mongodb",
                  config={
                      "database": "<database>",
                      "collection": "<collection>",
                      "uri": "<uri>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "mongodb",
      "config": {
          "database": "<database>",
          "collection": "<collection>",
          "uri": "<uri>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The name of the database on the target MongoDB instance.
</ParamField>

<ParamField type="string">
  The name of the collection within the database.
</ParamField>

<ParamField type="string">
  The instance connection string.
</ParamField>

## Learn more

* <Icon icon="blog" />  [How to go from S3 to MongoDB with no code using Unstructured](https://unstructured.io/blog/how-to-go-from-s3-to-mongodb-with-no-code-using-unstructured)


# MotherDuck
Source: https://docs.unstructured.io/api-reference/workflow/destinations/motherduck



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to MotherDuck.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A [MotherDuck account](https://app.motherduck.com).

* A [MotherDuck access token](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck/#creating-an-access-token) for the account.

* A database in the account.

  * [Create a database](https://motherduck.com/docs/sql-reference/motherduck-sql-reference/create-database/).
  * [List available databases](https://motherduck.com/docs/key-tasks/database-operations/basics-operations/#listing-databases).

  You can run commands to manage MotherDuck databases, schemas, tables, and more in the
  [MotherDuck UI](https://motherduck.com/docs/getting-started/motherduck-quick-tour/) or for example by connecting to MotherDuck with the
  [DuckDB CLI](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/connecting-to-motherduck/).

* A schema in the target database.

  * [Create a schema](https://duckdb.org/docs/sql/statements/create_schema.html).
  * You can list available schemas and their parent catalogs by running the following command in the MotherDuck UI or the DuckDB CLI:

    ```sql theme={null}
    SELECT * FROM information_schema.schemata;
    ```

  The MotherDuck connector uses the default schema name of `main` if not otherwise specified.

* A table in the target schema.

  * [Create a table](https://duckdb.org/docs/sql/statements/create_table).
  * You can list available tables in a schema by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog and schema names:

    ```sql theme={null}
    USE <catalog_name>.<schema_name>;
    SHOW TABLES;
    ```

  The MotherDuck connector uses the default table name of `elements` if not otherwise specified.

  For maximum compatibility, Unstructured recommends the following table schema:

  ```sql theme={null}
  CREATE TABLE elements (
      id VARCHAR,
      element_id VARCHAR,
      text TEXT,
      embeddings FLOAT[],
      type VARCHAR,
      system VARCHAR,
      layout_width DECIMAL,
      layout_height DECIMAL,
      points TEXT,
      url TEXT,
      version VARCHAR,
      date_created INTEGER,
      date_modified INTEGER,
      date_processed DOUBLE,
      permissions_data TEXT,
      record_locator TEXT,
      category_depth INTEGER,
      parent_id VARCHAR,
      attached_filename VARCHAR,
      filetype VARCHAR,
      last_modified TIMESTAMP,
      file_directory VARCHAR,
      filename VARCHAR,
      languages VARCHAR[],
      page_number VARCHAR,
      links TEXT,
      page_name VARCHAR,
      link_urls VARCHAR[],
      link_texts VARCHAR[],
      sent_from VARCHAR[],
      sent_to VARCHAR[],
      subject VARCHAR,
      section VARCHAR,
      header_footer_type VARCHAR,
      emphasized_text_contents VARCHAR[],
      emphasized_text_tags VARCHAR[],
      text_as_html TEXT,
      regex_metadata TEXT,
      detection_class_prob DECIMAL,
      partitioner_type VARCHAR
  );
  ```

  You can list the schema of a table by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog, schema, and table names:

  ```sql theme={null}
  USE <catalog_name>.<schema_name>;
  DESCRIBE TABLE <table_name>;
  ```

## Examples

To create a MotherDuck destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="motherduck",
                  config={
                      "database": "<database>",
                      "db_schema": "<db-schema>",
                      "table": "<table>",
                      "md_token": "<md-token>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "motherduck",
      "config": {
          "database": "<database>",
          "db_schema": "<db-schema>",
          "table": "<table>",
          "md_token": "<md-token>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The name of the target MotherDuck database.
</ParamField>

<ParamField type="string">
  The name of the target schema within the database.
</ParamField>

<ParamField type="string">
  The name of the target table within the schema.
</ParamField>

<ParamField type="string">
  The access token value within the MotherDuck account that has the appropriate access to the target database, schema, and table.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Unstructured's New MotherDuck Integration](https://unstructured.io/blog/unstructured-s-new-motherduck-integration)


# Neo4j
Source: https://docs.unstructured.io/api-reference/workflow/destinations/neo4j



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Neo4j.

## Requirements

You will need:

* A [Neo4j deployment](https://neo4j.com/deployment-center/).

  * For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local Neo4j deployments are not supported.
  * For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local Neo4j deployments are supported.

  The following video shows how to set up a Neo4j Aura deployment:

  <iframe title="YouTube video player" />

* The username and password for the user who has access to the Neo4j deployment. The default user is typically `neo4j`.

  * For a Neo4j Aura instance, the defaut user's is typically set when the instance is created.
  * For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, the default user is typically set during the deployment process.
  * For a local Neo4j deployment, you can [set the default user's initial password](https://neo4j.com/docs/operations-manual/current/configuration/set-initial-password/) or [recover an admin user and its password](https://neo4j.com/docs/operations-manual/current/authentication-authorization/password-and-user-recovery/).

* The connection URI for the Neo4j deployment, which starts with `neo4j://`, `neo4j+s://`, `bolt://`, or `bolt+s://`; followed by `localhost` or the host name; and sometimes ending with a colon and the port number (such as `:7687`). For example:

  * For a Neo4j Aura deployment, browse to the target Neo4j instance in the Neo4j Aura account and click **Connect > Drivers** to get the connection URI, which follows the format `neo4j+s://<host-name>`. A port number is not used or needed.
  * For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, see
    [Neo4j on AWS](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-aws/),
    [Neo4j on Azure](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-azure/), or
    [Neo4j on GCP](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-gcp/)
    for details about how to get the connection URI.
  * For a local Neo4j deployment, the URI is typically `bolt://localhost:7687`
  * For other Neo4j deployment types, see the deployment provider's documentation.

  [Learn more](https://neo4j.com/docs/browser-manual/current/operations/dbms-connection).

* The name of the target database in the Neo4j deployment. A default Neo4j deployment typically contains two standard databases: one named `neo4j` for user data and another
  named `system` for system data and metadata. Some Neo4j deployment types support more than these two databases per deployment;
  Neo4j Aura instances do not.

  * [Create additional databases](https://neo4j.com/docs/operations-manual/current/database-administration/standard-databases/create-databases/)
    for a local Neo4j deployment that uses Enterprise Edition; or for Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP deployments.
  * [Get a list of additional available databases](https://neo4j.com/docs/operations-manual/current/database-administration/standard-databases/listing-databases/)
    for a local Neo4j deployment that uses Enterprise Edition; or for Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP deployments.

## Graph Output

The graph output of the Neo4j destination connector is represented in the following diagram:

```mermaid theme={null}
graph BT
    subgraph dn [Document Node]
    D[Document]
    end
    style dn stroke-dasharray: 5
    
    subgraph en [Element Nodes]
    UE1[UnstructuredElement]
    UE2[UnstructuredElement]
    UE3[UnstructuredElement]
    UE4[UnstructuredElement]
    UE5[UnstructuredElement]
    UE6[UnstructuredElement]
    end
    style en stroke-dasharray: 5
    
    UE1 -->|PART_OF_DOCUMENT| D
    UE2 -->|PART_OF_DOCUMENT| D
    UE3 -->|PART_OF_DOCUMENT| D
    UE4 -->|PART_OF_DOCUMENT| D
    UE5 -->|PART_OF_DOCUMENT| D
    UE6 -->|PART_OF_DOCUMENT| D

    subgraph cn [Chunk Nodes]
    C1[Chunk]
    C2[Chunk]
    C3[Chunk]
    C4[Chunk]
    end
    style cn stroke-dasharray: 5
    
    C1 -->|NEXT_CHUNK| C2
    C2 -->|NEXT_CHUNK| C3
    C3 -->|NEXT_CHUNK| C4

    C1 -->|PART_OF_DOCUMENT| D
    C2 -->|PART_OF_DOCUMENT| D
    C3 -->|PART_OF_DOCUMENT| D
    C4 -->|PART_OF_DOCUMENT| D

    UE1 -.->|PART_OF_CHUNK| C1
    UE2 -.->|PART_OF_CHUNK| C1
    UE3 -.->|PART_OF_CHUNK| C2
    UE4 -.->|PART_OF_CHUNK| C3
    UE5 -.->|PART_OF_CHUNK| C4
    UE6 -.->|PART_OF_CHUNK| C4
```

[View the preceding diagram in full-screen mode](https://mermaid.live/view#pako:eNqFlN9vgjAQx_-Vps-6REEfeFiyFZYli7hskCyTxXS0ihFaU9oHo_7vq_IjgIzyxN330157d70TjDmh0IFbgQ8JeA4iBvSXq9_CQRhYuTxWGWUS-Br9KQC39pYOyki5VB5Tel2XS8H3dExwnmAh8NEBs4LohKA6hJfSOkJe7hh6k1XI9C4qlkpQUjK1Oh1UrUHVHlRng-p8QO1kgRqzoC8JxuPH8_vTR7BevqzdJQoXnh-cgVvf0wRYJsA2ATMTMP8f6FQz1tVEiWL7Vi3RpHBW5rRtWm3TbpmdnMbGnKIipb73FazRa-i_nXXAKvC9ZFWHuJfs6nrIUCVkKBIy1AjZpgTfGuWhwVRnnDT6ZFC3-vVpo0v6dKvRJH263eiRXh2OYEZFhndEj5nTlY6gTPSriaCjfwndYJXKCEbsolGsJP88shg6-onRERRcbRPobHCaa0sdCJbU3WHdbFmFHDD75jyrIUp2kotFMddu4-3yB3k-fcg).

In the preceding diagram:

* The `Document` node represents the source file.
* The `UnstructuredElement` nodes represent the source file's Unstructured `Element` objects, before chunking.
* The `Chunk` nodes represent the source file's Unstructured `Element` objects, after chunking.
* Each `UnstructuredElement` node has a `PART_OF_DOCUMENT` relationship with the `Document` node.
* Each `Chunk` node also has a `PART_OF_DOCUMENT` relationship with the `Document` node.
* Each `UnstructuredElement` node has a `PART_OF_CHUNK` relationship with a `Chunk` element.
* Each `Chunk` node, except for the "last" `Chunk` node, has a `NEXT_CHUNK` relationship with its "next" `Chunk` node.

Learn more about [document elements](/concepts/document-elements) and [chunking](/concepts/chunking).

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

```text theme={null}
MATCH path=(source)-[relationship]->(target)
RETURN path
```

Query for `Chunk` to `Document` relationships:

```text theme={null}
MATCH (chunk:Chunk)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN chunk, relationship, doc
```

Query for `UnstructuredElement` to `Document` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, relationship, doc
```

Query for `UnstructuredElement` to `Chunk` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
RETURN element, relationship, chunk
```

Query for `Chunk` to `Chunk` relationships:

```text theme={null}
MATCH (this:Chunk)-[relationship:NEXT_CHUNK]->(previous:Chunk)
RETURN this, relationship, previous
```

Query for `UnstructuredElement` to `Chunk` to `Document` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[ecrelationship:PART_OF_CHUNK]-(chunk:Chunk)-[cdrelationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, ecrelationship, chunk, cdrelationship, doc
```

Query for `UnstructuredElements` containing the text `jury`, and show their `Chunk` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE element.text =~ '(?i).*jury.*'
RETURN element, relationship, chunk
```

Query for the `Chunk` with the specified `id`, and show its `UnstructuredElement` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE chunk.id = '731508bf53637ce4431fe93f6028ebdf'
RETURN element, relationship, chunk
```

Additionally, for the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API's workflow operations](/api-reference/workflow/),
when a [Named entity recognition (NER)](/concepts/enriching/ner) DAG node is added to a custom workflow,
any recognized entities are output as `Entity` nodes in the graph.

This additional graph output of the Neo4j destination connector is represented in the following diagram:

```mermaid theme={null}
graph TD
    Chunk -->|HAS_ENTITY| Entity
    Entity -->|ENTITY_TYPE| Entity
```

In the preceding diagram:

* The `Chunk` node represents one of the source file's Unstructured `Element` objects, after chunking.
* The `Entity` node represents a recognized entity.
* A `Chunk` node can have `HAS_ENTITY` relationships with `Entity` nodes.
* An `Entity` node can have `ENTITY_TYPE` relationships with other `Entity` nodes.

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

```text theme={null}
MATCH path=(source)-[relationship]->(target)
RETURN path
```

Query for `Entity` to `Entity` relationships:

```text theme={null}
MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
RETURN child, relationship, parent
```

Query for `Entity` nodes containing the text `PERSON`, and show their `Entity` relationships:

```text theme={null}
MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id = 'PERSON'
RETURN child, relationship, parent
```

Query for `Entity` nodes containing the text `amendment`, and show their `Chunk` relationships:

```text theme={null}
MATCH (element:Chunk)-[relationship:HAS_ENTITY]->(entity:Entity)
WHERE entity.id =~ '(?i).*amendment.*'
RETURN element, relationship, entity
```

QUERY FOR `Entity` nodes containing the text `PERSON`, and show their `Entity` to `Entity` to `Chunk` relationships:

```text theme={null}
MATCH (chunk:Chunk)-[ccrelationship:HAS_ENTITY]-(child:Entity)-[cprelationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id =~ 'PERSON'
RETURN chunk, ccrelationship, child, cprelationship, parent
```

## Examples

To create a Neo4j destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="neo4j",
                  config={
                      "uri": "<uri>",
                      "database": "<database>",
                      "username": "<username>",
                      "password": "<password>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "neo4j",
      "config": {
          "uri": "<uri>",
          "database": "<database>",
          "username": "<username>",
          "password": "<password>",
          "batch_size": <batch-size>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The connection URI for the Neo4j deployment, which typically starts with `neo4j://`, `neo4j+s://`, `bolt://`, or `bolt+s://`; is followed by the host name; and ends with a colon and the port number (such as `:7473`, `:7474`, or `:7687`).
</ParamField>

<ParamField type="string">
  The name of the target database in the Neo4j deployment. A default Neo4j deployment typically contains a standard database named neo4j for user data.
</ParamField>

<ParamField type="string">
  The name of the user who has access to the Neo4j deployment. A default Neo4j deployment typically contains a default user named `neo4j`.
</ParamField>

<ParamField type="string">
  The password for the user.
</ParamField>

<ParamField type="integer">
  The maximum number of nodes or relationships to be transmitted per batch.
</ParamField>


# OneDrive
Source: https://docs.unstructured.io/api-reference/workflow/destinations/onedrive



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to OneDrive.

## Requirements

You will need:

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* The OneDrive and SharePoint Online plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.

  The following video shows how to get a path:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials, and a username and password. Both authentication types require a Microsoft Entra ID app registration.

  <Note>
    The OneDrive connector does not support any other authentication methods, such as multifactor (MFA) or passwordless authentication.
  </Note>

  You will need to provide the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration that has access to the target OneDrive account, and
  the app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Files.ReadWrite.All` (if both reading and writing are needed)
  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
  * `Directory.Read.All`

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* Both authentication types also require the User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

* For username and password authentication, you will also need the password for the target UPN.

## Examples

To create a OneDrive destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="onedrive",
                  config={
                      "client_id": "<client-id>",
                      "user_pname": "<user-pname>",
                      "password": "<password>",  # For username and password authentication
                      "tenant": "<tenant>",
                      "authority_url": "<authority-url>",
                      "client_cred": "<client-cred>",
                      "remote_url": "<remote-url>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "onedrive",
      "config": {
          "client_id": "<client-id>", 
          "user_pname": "<user-pname>",
          "password": "<password>", # For username and password authentication.
          "tenant": "<tenant>", 
          "authority_url": "<authority-url>",
          "client_cred": "<client-cred>",
          "remote_url": "<remote-url>" 
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account.
</ParamField>

<ParamField type="string">
  The User Principal Name (UPN) for the OneDrive user account in Entra ID. This is typically the user's email address.
</ParamField>

<ParamField type="string">
  Required for username and password authentication. The password for the target UPN.
</ParamField>

<ParamField type="string">
  The directory (tenant) ID of the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The authentication token provider URL for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The client secret for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  Source connector only. The path to the target folder in the OneDrive account, starting with the account's root folder, for example `my-folder/my-subfolder`.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to recursively access files from subfolders within the specified OneDrive `path`.
</ParamField>

<ParamField type="string">
  Destination connector only. `onedrive://`, followed by the path to the target folder in the OneDrive account, starting with the account's root folder, for example `onedrive://my-folder/my-subfolder`.
</ParamField>


# OpenSearch
Source: https://docs.unstructured.io/api-reference/workflow/destinations/opensearch



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to OpenSearch. This page covers configuration for both Amazon OpenSearch Service (managed domains) and Amazon OpenSearch Serverless collections.

## Requirements

You will need:

## Set up OpenSearch

Supported OpenSearch installations vary by product:

* **[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):** Non-local OpenSearch only. Local OpenSearch instances are not supported.
* **[Unstructured Ingest](/open-source/ingestion/overview):** Both local and non-local OpenSearch instances are supported.

### Set up an AWS OpenSearch Service domain

To set up an [AWS OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) domain, complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.

4. Click **Create domain**.

5. In the **Name** tile, for **Domain name**, enter some unique domain name for your new OpenSearch domain.

6. In the **Domain creation method** tile, select **Easy create**. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With **Standard create**, you must enable FGAC manually.) [Standard create method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html).

7. In the **Engine options** tile, for **Version**, AWS recommends that you select the latest version.

8. In the **Network** tile, for **Network**, select a network access method.
   For faster setup, this example uses the **Public access** method.
   [VPC access method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#prerequisites-vpc-endpoints).

9. For **IP address type**, select **Dual-stack mode**.

10. In the **Fine-grained access control** (FGAC) tile, do one of the following:

    * If you want to use an existing AWS IAM user in the AWS account as the domain's master user, then for **Master user**, select **Set IAM ARN as master user**. Then enter the IAM ARN for the master user in the **IAM ARN** box.
    * If you want to create a master user and password as the domain's master user instead, then for **Master user**, select **Create master user**. Then specify some username and password for this
      new master user by filling in the **Master username**, **Master password**, and **Confirm master password** fields. Make
      sure to save the master user's password in a secure location.

11. Click **Create**.

12. After the domain is created, you must allow Unstructured to access the domain, as follows:

    a. If the new domain's settings page is not already showing, open it as follows:
    in your Amazon OpenSearch Service console, on the sidebar, expand **Managed clusters**, and then click **Domains**. Then,
    in the list of available domains, click the name of the newly created domain.<br />
    b. On the **Security configuration** tab, click **Edit**.<br />
    c. In the **Access policy** tile, for **Domain access policy**, select **Only use fine-grained access control**.<br />
    d. Click **Clear policy**. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).<br />
    e. Click **Save changes**.

### Set up an Amazon OpenSearch Serverless collection

To set up an [Amazon OpenSearch Serverless collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-create-console.html), complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Serverless**, and then click **Dashboard**.

4. Click **Create collection**.

5. In the **Collection details** tile, for **Collection name**, enter some unique name for your new OpenSearch Serverless collection.
   Optionally, for **Description**, enter some meaningful description for your new collection.

6. For **Collection type**, select **Search**.

   <Note>
     Unstructured does not support the **Vector search** collection type. If you need vector search support, you can either continue
     with these steps to use the **Search** collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead.
     However, note that the Amazon OpenSearch Serverless **Search** collection type is not as optimal as the **Vector search** collection type.
   </Note>

7. In the **Collection creation method** tile, select **Standard create**.

8. For **Encryption**, choose an AWS KMS key type.

9. For **Network access settings**, choose an **Access type**.

10. For **Resource type**, select both **Enable access to OpenSearch endpoint** and **Enable access to OpenSearch Dashboards**.

11. Click **Next**.

12. In the **Definition method** tile, select **JSON**.

13. In the **JSON editor** box, enter the following JSON, replacing the following placeholders:

    * Replace `<collection-name>` with the name of the new OpenSearch Serverless collection.
    * Replace `<account-id>` with the target AWS account ID.
    * Replace `<user-id>` with the ID of the target AWS IAM user.

    ```json theme={null}
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    ```

14. Click **Next**.

15. For **Data access policy settings**, select **Create as a new data access policy**.

16. In the **Name and description** tile, enter some unique name and an optional description for the new data access policy.

17. Click **Next**.

18. Enter any desired index details, and click **Next** again. For example:

    a. For **Index name**, enter the name of the new index in the collection.<br />
    b. For **Automatic Semantic Enrichment fields**, click **Add**, enter `embeddings` for **Automatic Semantic Enrichment field name**, click **Add**, and click **Confirm**.<br />
    c. For **Lexical search fields**, click **Add**, enter `text` for **Field name** and select **Text** for **Data type**, click **Add**, and click **Confirm**.<br />

19. Click **Submit**.

### Set up a local OpenSearch instance

The following video shows how to set up a [local OpenSearch](https://opensearch.org/downloads.html) instance.

<iframe title="YouTube video player" />

### Get the host URL

Find your host URL in the AWS console using the steps for your OpenSearch type.

#### Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.
4. In the list of available domains, click the name of your domain.
5. In the **General information** tile, copy the value of **Domain endpoint v2 (dual stack)**.

#### Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Serverless**, and then click **Dashboard**.
4. In the list of available collections, click the name of your collection.
5. On the **Overview** tab, in the **Endpoint** tile, copy the value of **OpenSearch endpoint**.

#### Local OpenSearch instance

Your local instance URL depends on how OpenSearch is installed and configured. For guidance, see [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) in the OpenSearch documentation.

### Create a search index

The name of the search index on the instance is required.

For the destination connector, if you need to create an index and you're using a master user and password as the domain's master user, you can use for example the following `curl` command. Replace the following placeholders:

* Replace `<host>` with the instance's host URL.
* Replace `<port>` with the instance's port number, which is typically `443` (for encrypted connections, and less commonly `9200` for unencrypted connections).
* Replace `<master-username>` with the master user's name, and replace `<master-password>` with the master user's password.
* Replace `<index-name>` with the name of the new search index on the instance.
* Replace `<index-schema>` with the schema for the new search index on the instance. A schema is optional; see the explanation
  following this `curl` command for more information.

```bash theme={null}
curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
```

If you're using an existing AWS IAM user as the domain's master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the preceding `curl` command. To learn how, see [create-index](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-index.html) in the AWS CLI Command Reference.

For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

For objects in the `metadata` field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index's schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
required by Unstructured whenever you create your own index schema:

* `element_id`
* `record_id`, which is required by Unstructured for intelligent record updates.
* `type`, which is not required, but highly recommended.
* `text`
* `embeddings` if embeddings are generated; make sure to set `dimension` to the same number of dimensions as the embedding model generates.

```json theme={null}
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

See also:

* [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/)
* [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
* [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
* [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
* [Unstructured document elements and metadata](/concepts/document-elements)

### Set up master user authentication

<Note>
  If you are using Enterprise Connect on a [dedicated instance](/business/dedicated-instances/overview), you do not need **master user credentials**. Skip to the **Enterprise Connect** section on this page.
</Note>

For non-local OpenSearch instances, or if you're using basic authentication to a local OpenSearch instance, you will need the master user's name and password.

For local OpenSearch instances, if you're using certificates for authentication instead of basic authentication, you will need:

* The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
* The path to the combined private key and certificate file, or
* The paths to the separate private key and certificate files.

To learn more, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

## Examples

To create an OpenSearch destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="opensearch",
                  config={
                      "hosts": ["https://<host>:<port>"],
                      "index_name": "<index-name>",
                      "username": "<username>",
                      "password": "<password>",
                      "aws_access_key_id": "<aws-access-key-id>",
                      "aws_secret_access_key": "<aws-secret-access-key>",
                      "aws_session_token": "<aws-session-token>",
                      "use_ssl": <True|False>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "opensearch",
      "config": {
      "hosts": ["https://<host>:<port>"],
          "index_name": "<index-name>",
          "username": "<username>",
          "password": "<password>",
          "aws_access_key_id": "<aws-access-key-id>",
          "aws_secret_access_key": "<aws-secret-access-key>",
          "aws_session_token": "<aws-session-token>",
          "use_ssl": "true|false"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string[]">
  The OpenSearch instance's host URL, in the format `https://<host>:<port>`.
</ParamField>

<ParamField type="string">
  The name of the search index on the instance.
</ParamField>

<ParamField type="string">
  For basic authentication, the domain's master user's name.
</ParamField>

<ParamField type="string">
  For basic authentication, the domain's master user's password.
</ParamField>

<ParamField type="string">
  For AWS IAM user authentication, the AWS access key ID. For AWS STS authentication, a temporary AWS access key ID.
</ParamField>

<ParamField type="string">
  For AWS IAM user authentication, the AWS secret access key. For AWS STS authentication, a temporary AWS secret access key.
</ParamField>

<ParamField type="string">
  For AWS STS authentication, the temporary AWS STS session token.
</ParamField>

<Warning>
  If you are using the **IAM + Session Token** authentication method: AWS STS credentials (consisting of a temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding OpenSearch connector. You must get a new set of AWS STS credentials to replace the expired ones, which produces a new, refreshed temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token. For more information, see [Request temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html).

  After you generate refreshed temporary AWS STS credentials, you must update the OpenSearch connector's settings with the new, refreshed AWS STS credentials.
</Warning>

<ParamField type="string[]">
  Source connector only. Any specific fields to be accessed in the index.
</ParamField>

<ParamField type="boolean">
  Set to `true` if the OpenSearch instance requires an SSL connection; otherwise `false`.
</ParamField>

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an OpenSearch connector to use Enterprise Connect via the Unstructured API, first set up your AWS IAM role:

### Create the IAM role

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your OpenSearch resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

### Attach a permissions policy

To allow Unstructured to access your OpenSearch installation, you may need to attach a permissions policy to the IAM role. Whether it is required depends on your OpenSearch type and how your domain access policy is configured.

#### Amazon OpenSearch Service (managed)

A permissions policy is typically not required for a managed OpenSearch service domain that has the following:

* The domain's access policy is permissive (for example, `"Principal": "*"`)
* Fine-grained access control (FGAC) is the primary access control

Both conditions above are satisfied if you selected **Easy create** in step 6 (which enables FGAC by default) and followed step 12(d) (for the domain access policy) in the [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain) section. No identity-based permissions policy is needed, and the trust policy you attached in [Create the IAM role](#create-the-iam-role) is sufficient for Enterprise Connect to function. Skip to [Grant access to OpenSearch resources](#grant-access-to-opensearch-resources).

If either condition is not met and your domain's access policy explicitly restricts access to specific principals or actions, attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:<region>:<account-id>:domain/<domain-name>/*"
    }
  ]
}
```

| Placeholder     | Value                                                  |
| --------------- | ------------------------------------------------------ |
| `<region>`      | The AWS region of the domain, for example `us-east-1`. |
| `<account-id>`  | The 12-digit ID of your AWS account.                   |
| `<domain-name>` | The name of the OpenSearch domain.                     |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

#### Amazon OpenSearch Serverless

The OpenSearch Serverless connector requires a permissions policy. Attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "aoss:APIAccessAll",
      "Resource": "arn:aws:aoss:<region>:<account-id>:collection/<collection-id>"
    }
  ]
}
```

| Placeholder       | Value                                                      |
| ----------------- | ---------------------------------------------------------- |
| `<region>`        | The AWS region of the collection, for example `us-east-1`. |
| `<account-id>`    | The 12-digit ID of your AWS account.                       |
| `<collection-id>` | The ID of the OpenSearch Serverless collection.            |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

### Grant access to OpenSearch resources

After creating the IAM role and configuring AWS permissions, you must also authorize the role within OpenSearch itself. AWS authorization and OpenSearch authorization are separate systems.

The steps in this section configure the OpenSearch authorization for your IAM role. The process you follow depends on your OpenSearch type.

#### Amazon OpenSearch Service (managed) — Map the role inside OpenSearch

You only need to map the IAM role inside OpenSearch if fine-grained access control (FGAC) is enabled on your domain:

* If you selected **Easy create** in step 6 of [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain), FGAC was enabled automatically and these steps are required.
* If your domain has FGAC disabled, skip this section. Your IAM role setup is complete. Proceed to the next section to create your connector.

**Select or create an OpenSearch role**

An *OpenSearch role* is an internal security construct, distinct from an AWS IAM role. The IAM role ARN is mapped to an internal *OpenSearch role* by registering it as a **backend role**. This is how OpenSearch links an external AWS identity to an internal permission set. First, select or create the *OpenSearch role* that has permission to perform the actions your connector needs:

* For a quick proof-of-concept, use the built-in `all_access` *OpenSearch role*, already present in OpenSearch Dashboards under **Security** → **Roles**.
* For production, first [create a least-privilege *OpenSearch role*](https://opensearch.org/docs/latest/security/access-control/users-roles/) in OpenSearch Dashboards under **Security** → **Roles**, scoped to the specific indices your connector will access.

Once you have your *OpenSearch role*, complete the steps below to map your IAM role ARN to it. Skipping this mapping causes the connector to fail with a 403 error, even if the IAM role passes the AWS authorization checks:

```
security_exception ... no permissions for [indices:admin/aliases/get] and User [name=arn:aws:iam::...:role/...]
```

For more information about managing OpenSearch roles, see [Users and roles](https://opensearch.org/docs/latest/security/access-control/users-roles/) in the OpenSearch documentation.

**Map your IAM role ARN to the OpenSearch role**

You can add the mapping using OpenSearch Dashboards or the API:

* **OpenSearch Dashboards:** Go to **Security** → **Roles** → select the *OpenSearch role* → **Mapped users** tab → add the IAM role ARN under **Backend roles**. For more information, see [Fine-grained access control in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html) in the *AWS OpenSearch Service Developer Guide*.

* Using the **API**:

  ```bash theme={null}
  curl -XPUT "$DOMAIN/_plugins/_security/api/rolesmapping/<opensearch-role-name>" \
    -H "Content-Type: application/json" \
    -u "$ADMIN_USER:$ADMIN_PASS" \
    -d '{"backend_roles":["<assumed-role-arn>"]}'
  ```

  Replace `<opensearch-role-name>` with the name of the *OpenSearch role* that you selected or created in this section. Replace `<assumed-role-arn>` with the ARN of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role).

  <Note>
    `PUT` replaces the entire role mapping. To preserve existing backend roles, first `GET` the current mapping and include all existing entries in your `PUT` request.
  </Note>

#### Amazon OpenSearch Serverless — Add the role to a data access policy

OpenSearch Serverless does not use FGAC. Authorization is managed through data access policies attached to the collection.

<Note>
  OpenSearch Serverless limits each AWS account and region to 10 data access policies. Where possible, add the IAM role ARN to an existing policy rather than creating a new one.
</Note>

Use the AWS CLI to update the collection's data access policy:

```bash theme={null}
# 1. Find the data access policy for your collection
aws opensearchserverless list-access-policies \
  --type data \
  --region <region> \
  --resource "collection/<collection-name>"

# 2. Get the current policy and note the policyVersion value in the response
aws opensearchserverless get-access-policy \
  --type data \
  --name <policy-name> \
  --region <region>

# 3. Update the policy, adding the IAM role ARN to the Principal array
aws opensearchserverless update-access-policy \
  --type data \
  --name <policy-name> \
  --policy-version <version-from-step-2> \
  --region <region> \
  --policy file://policy.json
```

The `--policy-version` value comes from the `get-access-policy` command above (#2) and enables optimistic concurrency. The `--policy` document replaces the entire policy. Include all existing principals and rules, not just your addition.

Example `policy.json`, replacing the placeholder values:

```json theme={null}
[
  {
    "Rules": [
      {
        "Resource": ["collection/<collection-name>"],
        "Permission": ["aoss:DescribeCollectionItems"],
        "ResourceType": "collection"
      },
      {
        "Resource": ["index/<collection-name>/*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument",
          "aoss:UpdateIndex"
        ],
        "ResourceType": "index"
      }
    ],
    "Principal": ["arn:aws:iam::<account-id>:role/<role-name>"],
    "Description": "Unstructured connector access"
  }
]
```

| Placeholder         | Value                                                                                               |
| ------------------- | --------------------------------------------------------------------------------------------------- |
| `<collection-name>` | The name of the OpenSearch Serverless collection.                                                   |
| `<account-id>`      | The 12-digit ID of your AWS account.                                                                |
| `<role-name>`       | The name of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role). |

**For source-only connectors:** remove `aoss:WriteDocument`,  `aoss:UpdateIndex`, and `aoss:CreateIndex` from the `Permission` array.

<Note>
  Policy changes propagate within seconds, but may occasionally take up to a minute. If the connector returns a 403 immediately after updating the policy, wait a moment and retry before treating it as a configuration error.
</Note>

### Create the destination connector with Enterprise Connect

To create a destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="opensearch",
                  config={
                      "hosts": ["https://<host>:<port>"],
                      "index_name": "<index-name>",
                      "role_arn": "<role-arn>",
                      "external_id": "<external-id>",
                      "use_ssl": <True|False>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "opensearch",
      "config": {
          "hosts": ["https://<host>:<port>"],
          "index_name": "<index-name>",
          "role_arn": "<role-arn>",
          "external_id": "<external-id>",
          "use_ssl": "true|false"
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string[]">
  The OpenSearch instance's host URL, for example `https://<host>:<port>`.
</ParamField>

<ParamField type="string">
  The name of the search index on the instance.
</ParamField>

<ParamField type="string">
  The ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
</ParamField>

<ParamField type="string">
  The unique value you chose as your External ID. Must match the `sts:ExternalId` condition in the IAM role's trust policy.
</ParamField>

<ParamField type="boolean">
  Set to `true` if the OpenSearch instance requires an SSL connection; otherwise `false`.
</ParamField>


# Overview
Source: https://docs.unstructured.io/api-reference/workflow/destinations/overview



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

To use the [Unstructured API's workflow operations](/api-reference/workflow/overview) to manage destination connectors, do the following:

| Task                                           | Python function                   | cURL                                                                                         |
| ---------------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------- |
| Get a list of available destination connectors | `destinations.list_destinations`  | `GET` [`/destinations`](/api-reference/api/destination/list-destinations)                    |
| Get information about a destination connector  | `destinations.get_destination`    | `GET` [`/destinations/<connector-id>`](/api-reference/api/destination/get-destination)       |
| Create a destination connector                 | `destinations.create_destination` | `POST` [`/destinations`](/api-reference/api/destination/create-destination)                  |
| Update a destination connector                 | `destinations.update_destination` | `PUT` [`/destinations/<connector-id>`](/api-reference/api/destination/update-destination)    |
| Delete a destination connector                 | `destinations.delete_destination` | `DELETE` [`/destinations/<connector-id>`](/api-reference/api/destination/delete-destination) |

To create or update a destination connector, you must also provide settings that are specific to that connector.
For the list of specific settings, see:

| Connector                                                                                 | Python SDK                       | cURL                             |
| ----------------------------------------------------------------------------------------- | -------------------------------- | -------------------------------- |
| [Astra DB](/api-reference/workflow/destinations/astradb)                                  | `ASTRADB`                        | `astradb`                        |
| [Azure AI Search](/api-reference/workflow/destinations/azure-ai-search)                   | `AZURE_AI_SEARCH`                | `azure_ai_search`                |
| [Azure Blob Storage](/api-reference/workflow/destinations/azure-blob-storage)             | `AZURE`                          | `azure`                          |
| [Couchbase](/api-reference/workflow/destinations/couchbase)                               | `COUCHBASE`                      | `couchbase`                      |
| [Databricks Volumes](/api-reference/workflow/destinations/databricks-volumes)             | `DATABRICKS_VOLUMES`             | `databricks_volumes`             |
| [Delta Tables in Amazon S3](/api-reference/workflow/destinations/delta-table)             | `DELTA_TABLE`                    | `delta_table`                    |
| [Delta Tables in Databricks](/api-reference/workflow/destinations/databricks-delta-table) | `DATABRICKS_VOLUME_DELTA_TABLES` | `databricks_volume_delta_tables` |
| [Elasticsearch](/api-reference/workflow/destinations/elasticsearch)                       | `ELASTICSEARCH`                  | `elasticsearch`                  |
| [Google Cloud Storage](/api-reference/workflow/destinations/google-cloud)                 | `GCS`                            | `gcs`                            |
| [IBM FileNet](/api-reference/workflow/destinations/filenet)                               | `FILNET`                         | `filenet`                        |
| [IBM watsonx.data](/api-reference/workflow/destinations/ibm-watsonxdata)                  | `IBM_WATSONX_S3`                 | `ibm_watsonx_s3`                 |
| [Kafka](/api-reference/workflow/destinations/kafka)                                       | `KAFKA_CLOUD`                    | `kafka-cloud`                    |
| [Local](/api-reference/workflow/destinations/local)                                       |                                  |                                  |
| [Milvus](/api-reference/workflow/destinations/milvus)                                     | `MILVUS`                         | `milvus`                         |
| [MongoDB](/api-reference/workflow/destinations/mongodb)                                   | `MONGODB`                        | `mongodb`                        |
| [MotherDuck](/api-reference/workflow/destinations/motherduck)                             | `MOTHERDUCK`                     | `motherduck`                     |
| [Neo4j](/api-reference/workflow/destinations/neo4j)                                       | `NEO4J`                          | `neo4j`                          |
| [OneDrive](/api-reference/workflow/destinations/onedrive)                                 | `ONEDRIVE`                       | `onedrive`                       |
| [OpenSearch](/api-reference/workflow/destinations/opensearch)                             | `OPENSEARCH`                     | `opensearch`                     |
| [Pinecone](/api-reference/workflow/destinations/pinecone)                                 | `PINECONE`                       | `pinecone`                       |
| [PostgreSQL](/api-reference/workflow/destinations/postgresql)                             | `POSTGRES`                       | `postgres`                       |
| [Qdrant](/api-reference/workflow/destinations/qdrant)                                     | `QDRANT_CLOUD`                   | `qdrant-cloud`                   |
| [Redis](/api-reference/workflow/destinations/redis)                                       | `REDIS`                          | `redis`                          |
| [Snowflake](/api-reference/workflow/destinations/snowflake)                               | `SNOWFLAKE`                      | `snowflake`                      |
| [S3](/api-reference/workflow/destinations/s3)                                             | `S3`                             | `s3`                             |
| [Teradata](/api-reference/workflow/destinations/teradata-sql)                             | `TERADATA`                       | `teradata`                       |
| [Teradata Vector Store](/api-reference/workflow/destinations/teradata-vector-v2)          | `TERADATAVECTOR_V2`              | `teradatavector_v2`              |
| [VAST](/api-reference/workflow/destinations/vast)                                         | `VAST`                           | `vast`                           |
| [Weaviate](/api-reference/workflow/destinations/weaviate)                                 | `WEAVIATE`                       | `weaviate`                       |


# Pinecone
Source: https://docs.unstructured.io/api-reference/workflow/destinations/pinecone



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Pinecone.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A Pinecone account. [Get an account](https://app.pinecone.io/).

* A Pinecone API key in your Pinecone account. To create an API key, do the following:

  1. [Sign in to your Pinecone account and open the API keys page](https://app.pinecone.io/organizations/-/projects/-/keys).
  2. Click **+ API key**.
  3. For **API key name**, enter some descriptive name for the API key.
  4. Click **Create key**.
  5. Copy the generated API key to some secure location. You will not be able to access this API key again after you close the dialog.

* A Pinecone serverless index in your Pinecone account.

  Creating a serverless index on your own is optional.
  An index is not required to exist in advance.

  When you set up the connector, at runtime, the index behavior is as follows:

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * Your workflow must contain an embedder node, and the embedder node must specify the embedding model that Unstructured will use to generate the embeedings.
  * If an existing index name is specified,
    and the number of dimensions that Unstructured generates does not match the number of dimensions that are specified in the existing index's embedding settings, the run will fail.
    You must change the number of dimensions in your workflow's embedder node or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The
    new index's name will be `u<short-workflow-id>-<short-embedding-model-name>-<number-of-dimensions>`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * If an existing index name is specified, and Unstructured generates embeddings,
    but the number of dimensions that are generated does not match the existing index's embedding settings, the run will fail.
    You must change your Unstructured embedding settings or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The new index's name will be `unstructuredautocreated`.

  <Note>
    If you create a new index or use an existing one, Unstructured recommends that all records in the target index have a field
    named `record_id` with a string data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the index or, in some cases, the operation could fail altogether.
  </Note>

  To create a serverless index on your own, do the following:

  1. [Sign in to your Pinecone account and open the Create a new index page](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless).

  2. For **Enter index name**, enter some descriptive name for the index.

  3. For **Configuration**, select the check box labelled **Custom settings**, or click the tile labelled **Manual configuration**.

     <Warning>
       Do not click any of the other tiles, such as **text-embedding-3-large**. Clicking any of these other tiles will cause Pinecone to generate embeddings instead of
       having Unstructured generate them. If Pinecone generates embeddings instead of Unstructured, this could cause any related Unstructured workflows to fail.
     </Warning>

  4. For **Vector type**, select **Dense**.

  5. For **Dimension**, enter the number of dimensions for the embeddings that Unstructured will generate.

     <Warning>
       The number of dimensions that you enter here must match the number of dimensions for the embedding model that you use in any related Unstructured workflows or ingestion pipelines. If these numbers do not
       match in both places, this could cause any related Unstructured workflows or ingestion pipelines to fail.
     </Warning>

  6. For **Metric**, select **cosine**.

  7. Leave **Capacity mode** set to **Serverless**.

  8. You can leave **Cloud provider** and **Region** set to their default values, or you can select a cloud provider and region that is closest to you, if available.

  9. Click **Create index**.

* Within a Pinecone serverless index, custom [namespaces](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces) are supported but are not required.

## Examples

To create a Pinecone destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="pinecone",
                  config={
                      "index_name": "<index-name>",
                      "namespace": "<namespace-name>",
                      "api_key": "<api-key>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "pinecone",
      "config": {
          "index_name" "<index-name>", 
          "namespace": "<namespace-name>",
          "api_key": "<api-key>",
          "batch_size" <batch-size>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The name of the index in the Pinecone database. If no value is provided, see the beginning of this article for the behavior at run time.
</ParamField>

<ParamField type="string">
  The name of any custom namespace in the Pinecone index to upsert data into.
</ParamField>

<ParamField type="string">
  The Pinecone API key.
</ParamField>

<ParamField type="integer">
  The maximum number of records to transmit in a single batch.
</ParamField>


# PostgreSQL
Source: https://docs.unstructured.io/api-reference/workflow/destinations/postgresql



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to PostgreSQL.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local PostgreSQL installations are not supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local PostgreSQL installations are supported.

The following video shows how to set up [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):

<iframe title="YouTube video player" />

The following video shows how to set up [Azure Database for PostgreSQL](https://azure.microsoft.com/products/postgresql):

<iframe title="YouTube video player" />

* A PostgreSQL instance.

  * [Create an Amazon RDS for PostgreSQL instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html).
  * [Create an Azure Database for PostgreSQL server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-deploy-on-azure-free-account).
  * [Install PostgreSQL locally](https://www.postgresql.org/docs/current/tutorial-install.html).

* The host name and port number for the instance.

  * For Amazon RDS for PostgreSQL, learn how to [get the host name and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html#postgresql-endpoint).
  * For Azure Database for PostgreSQL, learn how to [get the host](https://learn.microsoft.com/azure/postgresql/flexible-server/quickstart-create-server#get-the-connection-information). The port number is `5432`.
  * For local PostgreSQL installations, these values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.
  * For other installation types, see your PostgreSQL provider's documentation.

* Depending on your network security requirements, you might need to allow access to your instance only from specific IP addresses.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

  To learn how to allow these IP address ranges, see your PostgreSQL provider's documentation, for example with
  [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) or
  [Azure Database for PostgreSQL](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-manage-firewall-portal#create-a-firewall-rule-after-server-is-created).

  <Note>
    For Amazon RDS for PostgreSQL, Amazon recommends that you set the instance's **Public access** setting to **No** by default, as this
    approach is more secure. This means that no
    resources can connect to the instance outside of the instance's associated Virtual Private Cloud (VPC) without extra configuration.
    [Learn more](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Hiding).
    [Access an Amazon RDS instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html).

    If you must enable public access, set the instance's **Public access** setting to **Yes**, and then adjust the instance's related
    security group to allow this access.
    [Learn how](https://repost.aws/en/questions/QUxemKa9u5TV6CmLiO-r5prg/lost-public-access-to-aws-rds-postgresql-instance).

    [Troubleshoot issues with connecting to Amazon RDS instances](https://repost.aws/knowledge-center/rds-connectivity-instance-subnet-vpc).
  </Note>

* A database in the instance.

  * For Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, the default database name is `postgres` unless a custom database name was specified during the instance creation process.
  * For local PostgreSQL installations, learn how to [create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* A table in the database. Learn how to [create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

  For the destination connector, the table must have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the fields `id`, `element_id`, `record_id`, `text`, (and `embeddings`, if you are using `pgvector` and generating vector embeddings), as follows.
  `type` is an optional field, but highly recommended.

  If you are using `pgvector` and generating vector embeddings, the number of dimensions (in this example, `1536`) must match the number of dimensions for the associated embedding model that you use in any related Unstructured workflows or ingestion pipelines.

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT
    );
    ```

    ```sql PostgreSQL with pgvector theme={null}
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536)
    );
    ```
  </CodeGroup>

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  For objects in the `metadata` field that Unstructured produces and that you want to store in PostgreSQL, you must create fields in your table's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```

    ```sql PostgreSQL with pgvector  theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "embeddings" is required if you are generating vector embeddings.
    --   If you are generating embeddings, the number of dimensions in "embeddings" 
    --   must match the number of dimensions for the associated embedding model 
    --   that you use in any related Unstructured workflows or ingestion pipelines.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536),
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```
  </CodeGroup>

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  See also:

  * [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
  * [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  The following video shows how to use the `psql` utility to connect to PostgreSQL, list databases, and list and create tables:

  <iframe title="YouTube video player" />

* A user in the database, and a password for the user.

  * For Amazon RDS for PostgreSQL, learn how to [create a user](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Roles.html).
  * For Azure Database for PostgreSQL, learn how to [create a user](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-create-users).
  * For local PostgreSQL installations, learn how to [create a user](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* Database access for the user.

  * For Amazon RDS for PostgreSQL, learn how to [control user access](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Access.html).
  * For Azure Database for PostgreSQL, learn how to [control user access](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For local PostgreSQL installations, learn how to [give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).
  * For other installation types, see your PostgreSQL provider's documentation.

## Examples

To create a PostgreSQL destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="postgres",
                  config={
                      "host": "<host>",
                      "database": "<database>",
                      "port": <port>
                      "username": "<username>",
                      "password": "<password>",
                      "table_name": "<table_name>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "postgres",
      "config": {
          "host": "<host>",
          "database": "<database>",
          "port": "<port>",
          "username": "<username>",
          "password": "<password>",
          "table_name": "<table_name>",
          "batch_size": <batch-size>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The host name.
</ParamField>

<ParamField type="string">
  The name of the database.
</ParamField>

<ParamField type="string">
  The port number.
</ParamField>

<ParamField type="string">
  The username.
</ParamField>

<ParamField type="string">
  The user's password.
</ParamField>

<ParamField type="string">
  The name of the table in the database.
</ParamField>

<ParamField type="integer">
  The maximum number of rows to transmit at a time.
</ParamField>

<ParamField type="string">
  Source connector only. The name of the ID column in the table.
</ParamField>

<ParamField type="string">
  Source connector only. Set one or more values, with each value representing the name of a column to process (including the specified `id_column` column). The default is all columns if not otherwise specified.
</ParamField>

## Learn more

* <Icon icon="blog" />  [PostgreSQL Integration in the Unstructured Platform](https://unstructured.io/blog/postgresql-integration-in-the-unstructured-platform)


# Qdrant
Source: https://docs.unstructured.io/api-reference/workflow/destinations/qdrant



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Qdrant.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only [Qdrant Cloud](https://qdrant.tech/documentation/cloud-intro/) is supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Qdrant Cloud,
  [Qdrant local](https://github.com/qdrant/qdrant), and [Qdrant client-server](https://qdrant.tech/documentation/quickstart/) are supported.

The following video shows how to set up Qdrant Cloud:

<iframe title="YouTube video player" />

* For Qdrant local, the path to the local Qdrant installation, for example: `/qdrant/local`

* For Qdrant client-server, the Qdrant server URL, for example: `http://localhost:6333`

* For Qdrant Cloud:

  * A [Qdrant account](https://cloud.qdrant.io/login).

  * A [Qdrant cluster](https://qdrant.tech/documentation/cloud/create-cluster/).

  * The cluster's URL. To get this URL, do the following:

    1. Sign in to your Qdrant Cloud account.
    2. On the sidebar, under **Dashboard**, click **Clusters**.
    3. Click the cluster's name.
    4. Note the value of the **Endpoint** field, for example: `https://<random-guid>.<region-id>.<cloud-provider>.cloud.qdrant.io`.

  * A [Qdrant API key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys).

* The name of the target [collection](https://qdrant.tech/documentation/concepts/collections) on the Qdrant local installation,
  Qdrant server, or Qdrant Cloud cluster.

  Qdrant requires the target collection to exist before Unstructured can write to the collection.
  The following example code demonstrates the use of the [Python Qdrant Client](https://pypi.org/project/qdrant-client/) to create
  a collection on a Qdrant Cloud cluster, configuring the collection for vectors with 3072 dimensions:

  ```python Python theme={null}
  from qdrant_client import QdrantClient, models
  import os

  client = QdrantClient(
      url=os.getenv("QDRANT_URL"),
      api_key=os.getenv("QDRANT_API_KEY")
  )

  client.create_collection(
      collection_name=os.getenv("QDRANT_COLLECTION"),
      vectors_config=models.VectorParams(
          size=3072,
          distance=models.Distance.COSINE
      )
  )

  collection = client.get_collection(
                   collection_name=os.getenv("QDRANT_COLLECTION")
               )

  print(f"The collection named '{os.getenv("QDRANT_COLLECTION")}' exists and " +
        f"has a status of '{collection.status}'.")
  ```

## Examples

To create a Qdrant destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="qdrant-cloud",
                  config={
                      "url": "<url>",
                      "collection_name": "<collection-name>",
                      "batch_size": <batch-size>,
                      "api_key": "<api-key>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "qdrant-cloud",
      "config": {
          "url": "<url>",
          "collection_name": "<collection-name>",
          "batch_size": "<batch-size>",
          "api_key": "<api-key>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Qdrant cluster's URL.
</ParamField>

<ParamField type="string">
  The name of the target collection on the Qdrant cluster.
</ParamField>

<ParamField type="string">
  The maximum number of records to transmit at a time.
</ParamField>

<ParamField type="string">
  The Qdrant API key.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Qdrant Integration in the Unstructured Platform](https://unstructured.io/blog/streamlining-rag-pipeline-development-qdrant-integration-in-the-unstructured-platform)
* <Icon icon="blog" />  [How to go from S3 to Qdrant with no code using Unstructured](https://unstructured.io/blog/how-to-go-from-s3-to-qdrant-with-no-code-using-unstructured)


# Redis
Source: https://docs.unstructured.io/api-reference/workflow/destinations/redis



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Redis.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A [Redis](https://redis.io) database, for example in [Redis Cloud](https://redis.io/cloud/).
* The target database's hostname and port number. [Create a database in Redis Cloud](https://redis.io/docs/latest/operate/rc/rc-quickstart/#create-an-account).
* The username and password for the target database. [Get the username and password in Redis Cloud](https://redis.io/docs/latest/operate/rc/rc-quickstart/#connect-to-a-database).
* The database number for the target database. Redis databases are typically numbered from 0 to 15, with the default database number typically being 0.

## Examples

To create a Redis destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="redis",
                  config={
                      "database": "<database>",
                      "ssl": <True|False>,
                      "batch_size": <batch-size>,
                      "key_prefix": "<key-prefix>",

                      # For URI authentication, include only "uri" key:
                      # "uri": "<uri>",

                      # For password authentication, include these keys instead:
                      # "host": "<host>",
                      # "port": <port>,
                      # "username": "<username>",
                      # "password": "<password>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "redis",
      "config": {
          "database": <database>,
          "ssl": <true|false>,
          "batch_size": <batch-size>,
          "key_prefix": "<key-prefix>",

          # For URI authentication:
          "uri": "<uri>"
        
          # For password authentication:
          "host": "<host>",
          "port": <port>,
          "username": "<username>",
          "password": "<password>"   
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  For URI authentication, the connection URI for the target database user and password, hostname, and port number. Use the format of `redis://<username>:<password>@<hostname>:<port-number>`. If SSL encryption is enabled for the database, use `rediss://` instead of `redis://`.
</ParamField>

<ParamField type="string">
  The target database's hostname. Required for password authentication.
</ParamField>

<ParamField type="integer">
  For password authentication, the database's port number.
</ParamField>

<ParamField type="string">
  The name of the database user (not the logged in user). Required for password authentication.
</ParamField>

<ParamField type="string">
  The database user's password. Required for password authentication.
</ParamField>

<ParamField type="integer">
  The number (index) for the target database.
</ParamField>

<ParamField type="string">
  A string to prepend to each element's `element_id`. This is useful for distinguishing between different data sets in the same Redis instance.
</ParamField>

<ParamField type="integer">
  The maximum number of records to upload in a single batch.
</ParamField>

<ParamField type="boolean">
  Set to `true` to enable SSL encryption for the connection. Set to `true` only if SSL is already set up and enabled for the target database.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Redis Support in Unstructured Platform: Supercharging Your RAG Pipeline](https://unstructured.io/blog/redis-support-in-unstructured-platform-supercharging-your-rag-pipeline)
* <Icon icon="blog" />  [Getting Started with Unstructured and Redis](https://unstructured.io/blog/getting-started-with-unstructured-and-redis)


# S3
Source: https://docs.unstructured.io/api-reference/workflow/destinations/s3



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Amazon S3.

## Requirements

You will need:

The following video shows how to fulfill the minimum set of Amazon S3 requirements:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
  [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

  <img alt="Enable anonymous bucket access" />

* For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
  can be granted in one of the following ways:

  * Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
    To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
    Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

    <iframe title="YouTube video player" />

  * Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
    Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
    [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
    [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
    a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

    <Warning>
      AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
      generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
      You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
      a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

      To overwrite the expired credentials with the new set:

      * For Unstructured Pipelines, manually update the AWS Key, AWS Secret Key, and STS Token fields in Unstructured Pipelines
        for the corresponding S3 [source](/pipelines/sources/s3) or [destination](/pipelines/destinations/s3) connector.
      * For the Unstructured API, use the Unstructured API's workflow operations to call the
        [update source](/api-reference/api/source/update-source) or
        [update destination](/api-reference/api/destination/update-destination) connector operation
        for the corresponding S3 [source](/api-reference/workflow/sources/s3) or
        [destination](/api-reference/workflow/destinations/s3) connector.
      * For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
        corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
    </Warning>

* If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## FIPS and ambient credentials

<Note>
  The following information applies to [Unstructured Business dedicated instance and in-VPC](/business/overview) deployments only.
</Note>

Unstructured Business accounts support the Federal Information Processing Standard (FIPS) for
Amazon S3. [Learn more about AWS support for FIPS](https://aws.amazon.com/compliance/fips/). Specifically,
when creating an S3 connector with the [Unstructured Pipelines](/pipelines/overview) or
[Unstructured API's workflow operations](/api-reference/workflow/overview), Unstructured Business dedicated instance and in-VPC deployments
support the use of `fips`-based
[S3 endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html) and, for authenticated access to
S3 FIPS buckets, S3 ambient credentials.

To use Unstructured Pipelines to set up an S3 source or destination connector
to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.
2. When creating the connector, for the S3 connector's **Bucket URI** field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).
3. For the **Authentication Method** field, select **Ambient Credentials**.
4. Check the box labeled **Use Ambient Credentials**.
5. Save and test the connector.

To use the Unstructured API's workflow operations to set up an S3 source or
destination connector to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.

2. When creating the connector,
   for the `config` parameter's `remote_url` field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).

3. For the `config` parameter, add an `ambient_credentials` field, and set its value to true. For example:

   <CodeGroup>
     ```python Python SDK theme={null}
     # ...
     config={
         "...": "...",
         "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
         "ambient_credentials": True
     }
     # ...
     ```

     ```bash curl theme={null}
     # ...
     {
         "...": "...", 
         "config": {
             "...": "...",
             "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
             "ambient_credentials": "true"
         }
     }
     # ...
     ```
   </CodeGroup>

4. Run your code to create the connector.

## Examples

To create a S3 destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="s3",
                  config={
                      # For AWS Access Key ID with AWS Secret Access Key authentication:
                      "key": "<key>",
                      "secret": "<secret>",

                      # For AWS STS token authentication (use instead of or in addition to above):
                      # "token": "<token>",

                      "remote_url": "<remote_url>",
                      "endpoint_url": "<endpoint-url>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "s3",
      "config": {
          # For AWS access key ID with AWS secret access key authentication:
          "key": "<key>", 
          "secret": "<secret>", 

          # For AWS STS token authentication:
          "token": "<token>", 
          "key": "<key>", 
          "secret": "<secret>", 

          "remote_url": "<remote_url>",
          "endpoint_url": "<endpoint-url>",
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  For AWS access key ID or AWS STS token authentication, the AWS access key ID for the authenticated IAM user.
</ParamField>

<ParamField type="string">
  For AWS access key ID or AWS STS token authentication, the AWS secret access key corresponding to the AWS access key ID.
</ParamField>

<ParamField type="string">
  For AWS STS token authentication, the AWS STS session token for temporary access.
</ParamField>

<ParamField type="string">
  A custom URL, if connecting to a non-AWS S3 bucket.
</ParamField>

<ParamField type="string">
  The S3 URI to the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket’s root) or `s3://my-bucket/my-folder/`.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to access subfolders within the bucket.
</ParamField>

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an S3 connector to use Enterprise Connect via the Unstructured API, first set up your AWS IAM role:

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your S3 resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

3. Attach a permissions policy to the role that grants access to your S3 bucket:

   * **Source connector:** at minimum `s3:GetObject`, `s3:ListBucket`
   * **Destination connector:** at minimum `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`

   Scope the policy to the specific bucket or prefix your connector will access.

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

### Create the destination connector with Enterprise Connect

To create a destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="s3",
                  config={
                      "remote_url": "<remote-url>",
                      "role_arn": "<role-arn>",
                      "external_id": "<external-id>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "s3",
      "config": {
          "remote_url": "<remote-url>",
          "role_arn": "<role-arn>",
          "external_id": "<external-id>"
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The S3 URI to the bucket or folder, for example `s3://my-bucket/` or `s3://my-bucket/my-folder/`.
</ParamField>

<ParamField type="string">
  The ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
</ParamField>

<ParamField type="string">
  The unique value you chose as your External ID. Must match the `sts:ExternalId` condition in the IAM role's trust policy.
</ParamField>


# Snowflake
Source: https://docs.unstructured.io/api-reference/workflow/destinations/snowflake



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Snowflake.

## Requirements

You will need:

* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.

  <iframe title="YouTube video player" />

  To get the identifier for the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
  3. On the **Account** tab, note the value of the **Account Identifier** field.

  Alternatively, the following Snowflake query returns the current account's identifier:

  ```text theme={null}
  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  ```

* A Snowflake user, which can be a service user (recommended) or a human user.

  To create a service user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Projects > Worksheets**.

  3. Click the **+** button to create a SQL worksheet.

  4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:

     * Replace `<service-user-name>` with some name for the service user.
     * Replace `<default-role-name>` with the name of any default role for the service user to use.

     ```sql theme={null}
     CREATE USER <service-user-name>
     DEFAULT_ROLE = "<default-role-name>"
     TYPE = SERVICE
     ```

  5. Click the arrow icon to run the worksheet, which creates the service user.

  6. To get their login name, on the navigation menu, click **Admin > Users & Roles**.

  7. On the **Users** tab, in the list of available users, click the name of the target user.

  8. In the **About** tile, note the **Login Name** for the user.

  To create a human user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & roles**.
  3. Click the **Users** tab.
  4. Click **+ User**.
  5. Follow the on-screen guidance to specify the user's settings.
  6. Click **Create User**.
  7. To get their login name, on the navigation menu, click **Admin > Users & Roles**.
  8. On the **Users** tab, in the list of available users, click the name of the target user.
  9. In the **About** tile, note the **Login Name** for the user.

* A programmatic access token (PAT) for the Snowflake user.

  <iframe title="YouTube video player" />

  To create a programmatic access token (PAT) for a user:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.

  3. On the **Users** tab, in the list of available users, click the name of the target user.

  4. In the **Programmatic access tokens** tile, click the **Generate new token** button.

  5. Follow the on-screen guidance to specify the PAT's settings.

     <Warning>
       You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later.
       Once this PAT expires, the connector will stop working.
       To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and
       update your connector's settings with your new PAT's value.

       Unstructured does not notify you when a PAT is about to expire or has already expired.
       You are responsible for tracking your PATs' expiration dates and taking corrective action before they expire.
     </Warning>

  6. Click **Generate**.

  7. Copy the generated PAT's value to a secure location, as you will not be able to access it again. If you lose this PAT's value,
     you will need to repeat this procedure to generate a new, replacement one.

  The PAT will not work unless the Snowflake account also has a valid
  [network rule](https://docs.snowflake.com/en/user-guide/network-rules) along with a valid
  [network policy](https://docs.snowflake.com/en/user-guide/network-policies) attached to that rule.
  The network rule must also be activated on the Snowflake account to begin taking effect.

  To create a valid network rule:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Rules**.

  3. Click **+ Network Rule**.

  4. Enter some name for the network rule.

  5. For **Type**, select **IPv4**.

  6. For **Mode**, select **Ingress**.

  7. For **Identifiers**, next to the magnifying glass icon, enter `0.0.0.0/0`, and then press **Enter**.

     <Note>
       The `0.0.0.0/0` value allows all IP addresses to access the Snowflake account.
       You can specify a more specific IP address range if you prefer. However, this more specific IP address range
       will apply to all users, including the user for which you created the PAT.
     </Note>

  8. Click **Create Network Rule**.

  To create a valid network policy, attaching the preceding network rule to this policy at the same time:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click **+ Network Policy**.
  4. Enter some name for the network policy.
  5. Make sure **Allowed** is selected.
  6. In the **Select rule** drop-down list, select the precedingnetwork rule to attach to this network policy.
  7. Click **Create Network Policy**.

  To activate the network rule in the account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click the name of the precedingnetwork policy to activate.
  4. In the policy's side panel, click the ellipsis (three dots) icon, and then click **Activate On Account**.
  5. Click **Activate policy**.

* (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake [user's login name (not username) and the user's password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
  This user must be a human user. Passwords are not supported for service users.

* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  <iframe title="YouTube video player" />

  * To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
  * To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
  * To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
  * To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
  * To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
  3. Click the **Roles** tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  ```text theme={null}
  SHOW ROLES;
  ```

  [Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).

* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

  <iframe title="YouTube video player" />

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:

  ```text theme={null}
  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  ```

* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

  <iframe title="YouTube video player" />

  To view a list of available databases in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  ```text theme={null}
  SHOW DATABASES;
  ```

* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  ```text theme={null}
  SHOW SCHEMAS;
  ```

  The following Snowflake query returns a list of available schemas for the database named `<database_name>` in the current account:

  ```text theme={null}
  SHOW SCHEMAS IN DATABASE <database_name>;
  ```

* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand **Tables**.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named `<schema_name>` in the datbase named
  `<database_name>` in the current account:

  ```text theme={null}
  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  ```

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the columns `ID`, `ELEMENT_ID`, and `RECORD_ID`. The columns `TEXT` and `TYPE` are optional, but highly recommended.
  If you are generating embeddings, then you must also include the column `EMBEDDINGS`.

  In the following `CREATE TABLE` statement, replace the following placeholders with the appropriate values:

  * `<database_name>`: The name of the target database in the Snowflake account.
  * `<schema_name>`: The name of the target schema in the database.
  * `<number-of-dimensions>`: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search,
    this value must match the number of dimensions that you plan to have Snowflake generate or search against.

  ```sql SQL theme={null}
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      ELEMENT_ID VARCHAR,
      RECORD_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>)
  );
  ```

  For objects in the `metadata` field that Unstructured produces and that you want to store in a Snowflake table, you must create columns in your table's schema that
  follow Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  ```sql theme={null}
  -- The ID, RECORD_ID, and ELEMENT_ID columns are required.
  -- TEXT and TYPE are not required but highly recommended.
  -- EMBEDDINGS is required if embeddings are being generated.
  -- All other "metadata" columns are optional.
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      RECORD_ID VARCHAR,
      ELEMENT_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
      IS_EXTRACTED VARCHAR,
      POINTS VARCHAR,
      SYSTEM VARCHAR,
      LAYOUT_WIDTH INTEGER,
      LAYOUT_HEIGHT INTEGER,
      FILETYPE VARCHAR,
      LANGUAGES ARRAY,
      PAGE_NUMBER VARCHAR,
      IMAGE_MIME_TYPE VARCHAR,
      FILENAME VARCHAR,
      URL VARCHAR,
      RECORD_LOCATOR VARCHAR
  );
  ```

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).

## Examples

To create a Snowflake destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="snowflake",
                  config={
                      "account": "<account>",
                      "user": "<user>",
                      "host": "<host>",
                      "port": <port>,
                      "database": "<database>",
                      "schema": "<schema>",
                      "role": "<role>",
                      "password": "<programmatic-access-token>",
                      "record_id_key": "<record-id-key>",
                      "table_name": "<table_name>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "snowflake",
      "config": {
          "account": "<account>",
          "user": "<user>",
          "host": "<host>",
          "port": <port>,
          "database": "<database>",
          "schema": "<schema>",
          "role": "<role>",
          "password": "<programmatic-access-token>",
          "record_id_key": "<record-id-key>",
          "table_name": "<table_name>",
          "batch_size": <batch-size>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The target Snowflake account's identifier.
</ParamField>

<ParamField type="string">
  The name of the Snowflake role that the user belongs to. This role must have the appropriate access to the target Snowflake warehouse, database, schema, and table.
</ParamField>

<ParamField type="string">
  The target Snowflake user's login name (not their username).
</ParamField>

<ParamField type="string">
  The user's programmatic access token (PAT).

  <Note>
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  </Note>
</ParamField>

<ParamField type="string">
  The hostname of the target Snowflake warehouse.
</ParamField>

<ParamField type="integer">
  The warehouse's port number.
</ParamField>

<ParamField type="string">
  The name of the target Snowflake database.
</ParamField>

<ParamField type="string">
  The name of the target Snowflake schema within the database.
</ParamField>

<ParamField type="string">
  The name of the target Snowflake table within the database's schema. For the destination connector, the default is `elements` if not otherwise specified.
</ParamField>

<ParamField type="string">
  Source connector only. An array of columns to fetch from the table. By default, all columns are fetched unless otherwise specified.
</ParamField>

<ParamField type="string">
  Source connector only. Required. The name of the column that uniquely identifies each record in the table.
</ParamField>

<ParamField type="string">
  Destination connector only. The name of the column that uniquely identifies each record in the table.
</ParamField>

<ParamField type="integer">
  The maximum number of rows to fetch for each batch.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Powering Enterprise RAG: Unstructured's New Snowflake Integration](https://unstructured.io/blog/powering-enterprise-rag-unstructured-s-new-snowflake-integration)


# Teradata
Source: https://docs.unstructured.io/api-reference/workflow/destinations/teradata-sql



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Teradata.

## Requirements

You will need:

* A Teradata Vantage system that can be accessed by its host name or IP address.

  For example, a Teradata Vantage system in Teradata ClearScape Analytics Experience includes:

  * A Teradata ClearScape Analytics Experience account.
  * An environment in the account.
  * A Teradata Vantage database in the environment.
  * The name and password for a Teradata user who has the appropriate access to the database.

  [Learn how to create these in Teradata ClearScape Analytics Experience](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/).

* The system's corresponding host name or IP address.

  For example, you can get these values from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics Experience account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Host** value.<br />

* The name of the target database in the system. To get a list of available databases in the system, you can run a Teradata SQL query such as the following:

  ```sql theme={null}
  SELECT DatabaseName 
  FROM DBC.DatabasesV 
  ORDER BY DatabaseName;
  ```

* The name of the target table in the database. To get a list of available tables in a database, you can run a Teradata SQL query such as the following, replacing `<database-name>` with the name of the target database:

  ```sql theme={null}
  SELECT TableName
  FROM DBC.TablesV
  WHERE DatabaseName = '<database-name>' AND TableKind = 'T'
  ORDER BY TableName;
  ```

  When Unstructured writes rows to a table, the table's columns must have a schema that is compatible with Unstructured.
  Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances.
  This is because these schemas will vary based on
  your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
  any custom post-processing code that you run; and other factors.

  In any case, note the following about table schemas:

  * The following columns are always required by Unstructured: `record_id` and `element_id`.
  * The following columns are optional for Unstructured, but highly recommended: `text` and `type`.
  * The rest of the columns are optional and typically will be output by Unstructured as part of the `metadata` field.
  * If Unstructured is generating vector embeddings, the `embeddings` column is also required.

    <Warning>
      The destination connector outputs Unstructured-generated [embeddings](/concepts/embedding) that are not directly compatible
      with Teradata Enterprise Vector Store. To use embeddings with Teradata Enterprise Vector Store, Unstructured
      recommends that you choose from among the following options:

      * Define a column in your target table named `embeddings` that is of type `VARCHAR(64000)`, to store the
        Unstructured-generated embeddings. After Unstructured adds its embeddings to your `embeddings` column,
        choose from among Teradata's options to convert the `embeddings` column's `VARCHAR` values to the
        Teradata [VECTOR Data Type](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/VECTOR-Data-Type)
        yourself.
      * Omit any columns named `embedding`, `message`, or `num_tokens` from your target table. Then choose from
        among Teradata's options (such as [AI\_TextEmbeddings](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/In-Database-Analytic-Functions-for-Vector-Store))
        to have Teradata generate the embeddings for you, instead of having Unstructured generate them.
    </Warning>

  Here is an example table schema that is compatible with Unstructured. It includes all of the required and recommended columns, as
  well as a few additional columns that are typically output by Unstructured as part of the `metadata` field. Be sure to replace
  `<database-name>` with the name of the target database and `<table-name>` with the name of the target table (by Unstructured convention,
  the table name is typically `elements`, but this is not a requirement).

  ```sql theme={null}
  CREATE SET TABLE "<database-name>"."<table-name>" (
    "id" VARCHAR(64) NOT NULL,
    PRIMARY KEY ("id"),
    "record_id" VARCHAR(64),
    "element_id" VARCHAR(64),
    "text" VARCHAR(32000) CHARACTER SET UNICODE,
    "type" VARCHAR(50),
    "embeddings" VARCHAR(64000), -- Add this column only if Unstructured is generating vector embeddings.
    "last_modified" VARCHAR(50),
    "languages" VARCHAR(200),
    "file_directory" VARCHAR(500),
    "filename" VARCHAR(255),
    "filetype" VARCHAR(50),
    "record_locator" VARCHAR(1000),
    "date_created" VARCHAR(50),
    "date_modified" VARCHAR(50),
    "date_processed" VARCHAR(50),
    "permissions_data" VARCHAR(1000),
    "filesize_bytes" INTEGER,
    "parent_id" VARCHAR(64)
  )
  ```

* For the source connector, the name of the primary key column in the table (for example, a column named `id`, typically defined as `"id" VARCHAR(64) NOT NULL, PRIMARY KEY ("id")`).

* For the source connector, the names of any specific columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* For the destination connector, the name of the column in the table that uniquely identifies each record for Unstructured to perform any necessary record updates. By default convention, Unstructured expects this field to be named `record_id`.

* The name of the Teradata user who has the appropriate access to the target database.

  For example, you can get this from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Username** value.<br />

  The Teradata SQL command to get a list of available users is as follows:

  ```sql theme={null}
  SELECT UserName 
  FROM DBC.UsersV 
  ORDER BY UserName;
  ```

* The password for the user, which was set up when the user was created.

  If the user has forgotten their password, the Teradata SQL command to change a user's password is as follows, replacing `<user-name>` with the name of the user and `<new-password>` with the new password:

  ```sql theme={null}
  MODIFY USER <user-name> SET PASSWORD = '<new-password>';
  ```

  To change a user's password, you must be an administrator (such as the `DBC` user or another user with `DROP USER` privileges).

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

## Examples

To create a Teradata destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="teradata",
                  config={
                      "host": "<host>",
                      "database": "<database>",
                      "table_name": "<table-name>",
                      "batch_size": <batch-size>,
                      "record_id_key": "<record-id-key>",
                      "user": "<user>",
                      "password": "<password>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "teradata",
      "config": {
          "host": "<host>",
          "database": "<database>",
          "table_name": "<table-name>",
          "batch_size": <batch-size>,
          "record_id_key": "<record-id-key>",
          "user": "<user>",
          "password": "<password>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The hostname or IP address associated with the target Teradata Vantage database.
</ParamField>

<ParamField type="string">
  The name of the target database. By default, the default database name is used if not otherwise specified. To get the name of the default database, you can run the Teradata SQL command `SELECT DATABASE;`.
</ParamField>

<ParamField type="string">
  The name of the target table in the database.
</ParamField>

<ParamField type="integer">
  The maximum number of rows per batch.
</ParamField>

<ParamField type="string">
  Source connector only. Required. The name of the primary key column that Teradata uses to uniquely identify each record in the table.
</ParamField>

<ParamField type="string">
  Destination connector only. The name of the column that Unstructured uses to uniquely identify each record in the table for record update purposes.
</ParamField>

<ParamField type="string">
  Source connector only. An array of column names to fetch from the table. By default, all columns are fetched unless otherwise specified.
</ParamField>

<ParamField type="string">
  The name of the user who has the appropriate access to the database.
</ParamField>

<ParamField type="string">
  The password for the user.
</ParamField>


# Teradata Enterprise Vector Store
Source: https://docs.unstructured.io/api-reference/workflow/destinations/teradata-vector-v2

Send processed data from Unstructured to a Teradata Enterprise Vector Store.

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

[Teradata Enterprise Vector Store](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Introduction-to-the-Enterprise-Vector-Store-User-Guide) is a specialized database designed to store and manage high-dimensional vectors (numeric arrays that encode meaning or similarity).

Destination connectors support Version 2 (V2) of the Teradata Enterprise Vector Store APIs. For more information, see [Functional Reference of V2 APIs (Vector Collections)](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Functional-Reference-of-V2-APIs-Vector-Collections) in the *Teradata Enterprise Vector Store User Guide*.

## Requirements

You will need:

* The base URL for the Teradata Analytics REST API endpoint.

  The format depends on your **VantageCloud Enterprise** deployment type:

  * For private VPC: `https://<site-id>.private.cloud.teradata.com`
  * For public site gateway: `https://<site-id>.gateway.cloud.teradata.com`

  Replace `<site-id>` with your Teradata site ID. Contact your Teradata system administrator if you do not know your site ID.

  For more information, see [Vector Store Service Access and Authentication](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Setting-up-Vector-Store/Vector-Store-Service-Access-and-Authentication) in the *Teradata Enterprise Vector Store User Guide*.

  <Note>
    The REST API URL is not available for VantageCloud Lake (VCL) deployments.
    VCL deployments use JWT or PAT token-based authentication and are not currently supported by this connector.
  </Note>

* The username and password for your Teradata database account.

  This connector uses Basic Authentication (TD2), supported on AI-on-premises and VantageCloud Enterprise deployments.

  The account must have been granted the `TD_AIAdmin` role. This role is required to create vector collections. Contact your Teradata system administrator to confirm that your account has been granted this role.

  If you do not know your credentials, contact your Teradata system administrator.

  For more information, see [Authenticating Users](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Administering-Vector-Store/Authenticating-Users) in the *Teradata Enterprise Vector Store User Guide*.

* The name of the collection to create or add data to.

  A *vector collection* refers to an instance of Vector Store. An enterprise vector store may comprise thousands of vector collections, each tailored for a specific use case or workflow.

  Collection names must be unique within the vector store. They can only contain letters, numbers, and underscores, matching the following regex pattern: `^[a-zA-Z0-9_]+$`.

  * To use an existing collection, contact your Teradata system administrator for the collection name, or use the [Collection Manager](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Functional-Reference-of-V2-APIs-Vector-Collections/Using-Collection-Manager) to list available collections.
  * To create a new collection, provide a unique name. The connector creates the collection automatically on first use.

  For more information, see [Creating a Vector Collection](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Functional-Reference-of-V2-APIs-Vector-Collections/Creating-a-Vector-Collection) in the *Teradata Enterprise Vector Store User Guide*.

## Examples

The following examples show how to create a Teradata Enterprise Vector Store destination connector using the Python SDK or REST API.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="teradatavector_v2",
                  config={
                      "base_url": "<base-url>",
                      "user": "<user>",
                      "password": "<password>",
                      "host": "<host>",
                      "database": "<database>",
                      "collection_name": "<collection-name>",
                      "use_precomputed_embeddings": <True|False>,
                      "deduplicate": <True|False>
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "teradatavector_v2",
      "config": {
          "base_url": "<base-url>",
          "user": "<user>",
          "password": "<password>",
          "host": "<host>",
          "database": "<database>",
          "collection_name": "<collection-name>",
          "use_precomputed_embeddings": <true|false>,
          "deduplicate": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The base URL for the Teradata Analytics REST API endpoint. The format depends on your **VantageCloud Enterprise** deployment type:

  * For private VPC: `https://<site-id>.private.cloud.teradata.com`
  * For public site gateway: `https://<site-id>.gateway.cloud.teradata.com`

  Replace `<site-id>` with your Teradata site ID. Contact your Teradata system administrator if you do not know your site ID.

  For more information, see [Vector Store Service Access and Authentication](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Setting-up-Vector-Store/Vector-Store-Service-Access-and-Authentication) in the *Teradata Enterprise Vector Store User Guide*.
</ParamField>

<ParamField type="string">
  The username for your Teradata database account.

  The account must have been granted the `TD_AIAdmin` role. This role is required to create vector collections. Contact your Teradata system administrator to confirm that your account has been granted this role.
</ParamField>

<ParamField type="string">
  The password for your Teradata database account.
</ParamField>

<ParamField type="string">
  The database host. Defaults to the base URL if not otherwise specified.
</ParamField>

<ParamField type="string">
  The name of the database.
</ParamField>

<ParamField type="string">
  The name of the collection to create or add data to.

  Collection names must be unique within the vector store and can only contain letters, numbers, and underscores, matching the following regex pattern: `^[a-zA-Z0-9_]+$`.

  To create a new collection, provide a unique name. The connector creates the collection automatically on first use.
</ParamField>

<ParamField type="boolean">
  Set to `true` if your Unstructured workflow is configured to generate embedding vectors. Set to `false` to have Teradata generate embeddings automatically.
</ParamField>

<ParamField type="boolean">
  Set to `true` delete existing documents with the same filename before uploading new versions. This enables idempotent re-runs without destroying the entire collection. Set to `false` to preserve existing documents.
</ParamField>


# VAST
Source: https://docs.unstructured.io/api-reference/workflow/destinations/vast

Send processed data from Unstructured to [VAST](https://www.vastdata.com/).

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* A [VAST](https://www.vastdata.com/) user account with permissions to the bucket to which you want to connect.

* A bucket that contains the objects to which you want to connect. A bucket is a storage resource that uses Amazon's Simple Storage Service (S3) as an access protocol.

  To create a bucket, see [Creating S3 Buckets](https://kb.vastdata.com/documentation/docs/creating-s3-buckets-via-vms-1) in the *VAST Cluster Administrator's Guide*.

* The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/` (for example, `vast://my-bucket/`). If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/` (for example, `vast://my-bucket/my-folder/`).

* The URL for the VAST region in which the bucket is located (for example, `https://vast-cluster.my-company.com`).

* An enabled access and secret key pair that grants the authenticated user permission to the VAST bucket.

  To grant a user access to a bucket, and attach identity policies that define their permissions on that bucket, see [Managing S3 User Access from the VAST Web UI](https://kb.vastdata.com/documentation/docs/managing-s3-user-access-from-the-vast-web-ui-1) or [Managing S3 Access from the VAST CLI](https://kb.vastdata.com/documentation/docs/managing-s3-access-from-the-vast-cli) in the *VAST Cluster Administrator's Guide*.

  <Note>
    VAST recommends using identity policies and bucket policies to control S3 bucket access.

    * *Identity policies* are attached to users and groups, and defines the actions the users or groups can perform against buckets.
    * *Bucket policies* are attached to a view, and define actions that particular users or groups can perform against the view and files or directories under the view.

    For more information, see [Overview of S3 Access Management](https://kb.vastdata.com/documentation/docs/overview-of-s3-access-management) in the *VAST Cluster Administrator's Guide*.

    Identity and bucket policies support a subset of elements listed in [Amazon's IAM JSON Policy Reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html). For information about the required JSON format, supported elements and identity and bucket policy examples, see [Identity and Bucket Policy Reference](https://kb.vastdata.com/documentation/docs/identity-and-bucket-policy-reference-1) in the *VAST Cluster Administrator's Guide*.
  </Note>

  For source connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:ListBucket`
  * `s3:GetObject`

  For destination connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:PutObject`

VAST buckets are private by default. [Anonymous access](https://kb.vastdata.com/documentation/docs/overview-of-vast-cluster-s3-implementation-1#support-for-authenticated-or-anonymous-requests) can be explicitly enabled using bucket policies, but is not recommended.

## Examples

To create a VAST destination connector, see the following examples.
For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="vast",
                  config={
                      "remote_url": "<remote-ur>",
                      "endpoint_url": "<endpoint-url>",
                      "recursive": <true|false>,
                      "max-objects": <integer>,
                      "key": "<key>",
                      "secret": "<secret>" 
                  }
              )
          )
      )

      print(response.destination_connector_information)
  # ...
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "vast",
      "config": {
          "remote_url": "<remote-ur>",
          "endpoint_url": "<endpoint-url>",
          "recursive": <true|false>,
          "max-objects": <integer>,
          "key": "<key>",
          "secret": "<secret>" 
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/`. For example: `vast://my-bucket/`)

  If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/`. For example: `vast://my-bucket/my-folder/`

  <Note>
    For destination connectors, the relative directory structure of files is preserved when files are uploaded. Also, files are overwritten if they already exist at the destination.
  </Note>
</ParamField>

<ParamField type="boolean">
  Source connectors only. Set to `true` to access sub-folders within the bucket.
</ParamField>

<ParamField type="integer">
  Source connectors only. The maximum number of objects to index.
</ParamField>

<ParamField type="string">
  The VAST storage endpoint URL. For example, `https://vast-cluster.my-company.com`.
</ParamField>

<ParamField type="string">
  The access key ID for the authenticated user.
</ParamField>

<ParamField type="string">
  The secret access key corresponding to the preceding access key ID.
</ParamField>


# Weaviate
Source: https://docs.unstructured.io/api-reference/workflow/destinations/weaviate



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Weaviate.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview): only [Weaviate Cloud](https://weaviate.io/developers/wcs) clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview): Weaviate Cloud clusters,
  [Weaviate installed locally](https://weaviate.io/developers/weaviate/quickstart/local),
  and [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded) are supported.
* For Weaviate installed locally, you will need the name of the target collection on the local instance.
* For Embedded Weaviate, you will need the instance's connection URL and the name of the target collection on the instance.
* For Weaviate Cloud, you will need:

  <iframe title="YouTube video player" />

  * A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
    [Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).
  * The URL and API key for the database cluster. [Get the URL and API key](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).
  * The name of the target collection in the database. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).

    An existing collection is not required. At runtime, the collection behavior is as follows:

    For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview):

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your Weaviate cluster. If Unstructured generates embeddings,
      the new collection's name will be `U<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `U<short-workflow-id`.

    For [Unstructured Ingest](/open-source/ingestion/overview):

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your Weaviate cluster. The new collection's name will be `Unstructuredautocreated`.

    If Unstructured creates a new collection and generates embeddings, you will not see an `embeddings` property in tools such as the Weaviate Cloud
    **Collections** user interface. To view the generated embeddings, you can run a Weaviate GraphQL query such as the following. In this query, replace `<collection-name>` with
    the name of the new collection, and replace `<property-name>` with the name of each additional available property that
    you want to return results for, such as `text`, `type`, `element_id`, `record_id`, and so on. The embeddings will be
    returned in the `vector` property.

    ```text theme={null}
    {
      Get {
        <collection-name> {
          _additional {
            vector
          }
          <property-name>
          <property-name>
        }
      }
    }
    ```

If [auto-schema](https://docs.weaviate.io/weaviate/config-refs/collections#auto-schema) is enabled in Weaviate (which it is by default),
Weaviate can infer missing properties and add them to the collection definition at run time. However, it is a Weaviate best practice to manually define as much
of the data schema in advance as possible, since manual definition gives you the most control.

The minimum viable schema for Unstructured includes only the `element_id` and `record_id` properties. The `text` and `type` properties should also be included, but they are technically optional.
If you are using Unstructured to generate embeddings, you must

The following code example shows how to use the [weaviate-client](https://pypi.org/project/weaviate-client/) Python package to create a
collection in a Weaviate Cloud database cluster with this minimum viable schema, and to specify that Unstructured will generate the embeddings for this collection.
To connect to a locally hosted Weaviate instance instead, call [weaviate.connect\_to\_local](https://docs.weaviate.io/weaviate/connections/connect-local).
To connect to Embedded Weaviate instead, call [weaviate.connect\_to\_embedded](https://docs.weaviate.io/weaviate/connections/connect-embedded).

```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
```

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

For objects in the `metadata` field that Unstructured produces and that you want to store in a Weaviate collection, be sure to follow
Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You could create corresponding properties in your collection's schema by using the following property names and data types:

```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
        wvc.Property(
            name="metadata",
            data_type=wvc.DataType.OBJECT,
            nested_properties=[
                wvc.Property(name="is_extracted", data_type=wvc.DataType.TEXT),
                wvc.Property(
                    name="coordinates",
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(name="points", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="system", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="layout_width", data_type=wvc.DataType.NUMBER),
                        wvc.Property(name="layout_height", data_type=wvc.DataType.NUMBER),
                    ],
                ),
                wvc.Property(name="filetype", data_type=wvc.DataType.TEXT),
                wvc.Property(name="languages", data_type=wvc.DataType.TEXT_ARRAY),
                wvc.Property(name="page_number", data_type=wvc.DataType.TEXT),
                wvc.Property(name="image_mime_type", data_type=wvc.DataType.TEXT),
                wvc.Property(name="filename", data_type=wvc.DataType.TEXT),
                wvc.Property(
                    name="data_source",
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(name="url", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="record_locator", data_type=wvc.DataType.TEXT),
                    ],
                ),
                wvc.Property(
                    name="entities", 
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(
                            name="items", 
                            data_type=wvc.DataType.OBJECT_ARRAY,
                            nested_properties=[
                                wvc.Property(name="entity", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="type", data_type=wvc.DataType.TEXT),
                            ],
                        ),
                        wvc.Property(
                            name="relationships", 
                            data_type=wvc.DataType.OBJECT_ARRAY,
                            nested_properties=[
                                wvc.Property(name="to", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="from", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="relationship", data_type=wvc.DataType.TEXT),
                            ],
                        ),
                    ],
                ),
            ],
        ),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
```

Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

See also:

* [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
* [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

## Examples

To create a Weaviate destination connector, see the following examples.

For more information on working with destination connectors using the Unstructured API, see [Destination endpoints](/api-reference/api/destination/destination-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateDestinationRequest
  from unstructured_client.models.shared import CreateDestinationConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.destinations.create_destination(
          request=CreateDestinationRequest(
              create_destination_connector=CreateDestinationConnector(
                  name="<name>",
                  type="weaviate-cloud",
                  config={
                      "cluster_url": "<host-url>",
                      "collection": "<class-name>",
                      "api_key": "<api-key>"
                  }
              )
          )
      )

      print(response.destination_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/destinations" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "weaviate",
      "config": {
          "host_url": "<host-url>",
          "class_name": "<class-name>",
          "api_key": "<api-key>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The URL of the Weaviate database cluster.
</ParamField>

<ParamField type="string">
  The name of the target collection within the cluster. If no value is provided, see the beginning of this article for the behavior at run time.
</ParamField>

<ParamField type="string">
  The API key provided by Weaviate to access the cluster.
</ParamField>


# Errors
Source: https://docs.unstructured.io/api-reference/workflow/errors



For the [Unstructured Python SDK](/api-reference/workflow/overview#unstructured-python-sdk),
the [Unstructured API's workflow operations](/api-reference/workflow/overview) returns errors primarily through
the `UnstructuredClientError` class (the base class for all errors raised by the Unstructured Python SDK) and
the `HTTPValidationError` class (inherited from `UnstructuredClientError`). Less common errors are returned through the following classes:

* `httpx.RequestError`, the base class for request errors.
* `httpx.ConnectError`, for HTTP connection request errors.
* `httpx.TimeoutException`, for HTTP request timeout errors.
* `ServerError` (inherited from `UnstructuredClientError`), for server-side errors.
* `ResponseValidationError` (inherited from `UnstructuredClientError`), for type mismatches between the response data and the expected Pydantic model.

Each of the preceding classes has the following members:

| Member         | Type             | Description                                                        |
| -------------- | ---------------- | ------------------------------------------------------------------ |
| `message`      | `str`            | The eror message.                                                  |
| `status_code`  | `int`            | The HTTP response status code, for example `401`.                  |
| `headers`      | `httpx.Headers`  | A collection of HTTP response headers.                             |
| `body`         | `str`            | The HTTP body. This can be an empty string if no body is returned. |
| `raw_response` | `httpx.Response` | The raw HTTP response.                                             |

The following example shows how to handle the preceding errors. In this example,
a required Unstructured API key is intentionally commented out of the code, so that a
`401 Unauthorized` error is intentionally thrown.

```python theme={null}
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import ListWorkflowsRequest
from unstructured_client.models.errors import (
    UnstructuredClientError,
    HTTPValidationError
)
from unstructured_client.models.errors.servererror import ServerError
from unstructured_client.models.errors.responsevalidationerror import ResponseValidationError
import httpx

try:
    client = UnstructuredClient(
        # For example, intentionally leave out the API key to intentionally throw an error.
        # api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    )

    response = client.workflows.list_workflows(
        request=ListWorkflowsRequest()
    )
    
    print(f"Found {len(response.response_list_workflows)} workflows.")

except HTTPValidationError as e:
    print("Validation error (HTTP 422):", e)
except ServerError as e:
    print("Server error (HTTP 5XX):", e)
except ResponseValidationError as e:
    print("Response validation/type mismatch:", e)
except UnstructuredClientError as e:
    # This catches any other UnstructuredClientError not already caught above.
    # This and all of the other error classes in this example expose the following members:
    print("Other Unstructured client error:")
    print(f"Message:      {e.message}")
    print(f"Status code:  {e.status_code}")
    print(f"Body:         {e.body}")
    print(f"Raw response: {e.raw_response}")
    print(f"Headers:")

    for header in e.headers.raw:
        key = header[0].decode('utf-8')
        value = header[1].decode('utf-8')
        print(f"              {key}: {value}")

except httpx.ConnectError as e:
    print("HTTP connection error:", e)
except httpx.TimeoutException as e:
    print("HTTP timeout error:", e)
except httpx.RequestError as e:
    # This catches catch-all network errors from HTTP not already caught above.
    print("Other HTTPX request error:", e)
except Exception as e:
    # Optional: this catches any other unforeseen errors.
    print("Unexpected error:", e)
```

The results of running the preceding code are similar to the following:

```text theme={null}
Message:      API error occurred: Status 401. Body: {"detail":"API key is missing, please provide an API key in the header."}
Status code:  401
Body:         {"detail":"API key is missing, please provide an API key in the header."}
Raw response: <Response [401 Unauthorized]>
Headers:
              date: <date-and-time-of-the-error>
              server: <server-identifier>
              content-length: 73
              content-type: application/json
```


# Jobs
Source: https://docs.unstructured.io/api-reference/workflow/jobs

Manage Unstructured workflow jobs with the Python SDK or REST API. List jobs, monitor processing status, inspect failed files, and cancel running jobs.

A workflow job is created automatically whenever:

* A workflow runs on a schedule. For more details see [Create workflow](/api-reference/api/workflow/create-workflow).
* You run a workflow manually. For more details see [Run workflow](/api-reference/api/workflow/run-workflow).

To use the [Unstructured API's workflow operations](/api-reference/workflow/overview) to manage jobs, do the following:

| Task                                                                                                                                                           | Python function         | curl                                                                               |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------- |
| Get a list of available jobs                                                                                                                                   | `jobs.list_jobs`        | `GET` [`/jobs`](/api-reference/api/job/list-jobs)                                  |
| Get basic information about a job                                                                                                                              | `jobs.get_job`          | `GET` [`/jobs/<job-id>`](/api-reference/api/job/get-job)                           |
| Get information about a job's current processing status                                                                                                        | `jobs.get_job_details`  | `GET` [`/jobs/<job-id>/details`](/api-reference/api/job/get-job-details)           |
| Get the list of any failed files for a job and why those files failed                                                                                          | `jobs.get_failed_files` | `GET` [`/jobs/<job-id>/failed-files`](/api-reference/api/job/get-job-failed-files) |
| Run a workflow that takes one or more local files only as input, and the workflow exists only for the duration of that job's run (known as an *on-demand job*) |                         | `POST` [`/jobs/`](/api-reference/api/job/create-job)                               |
| Cancel a running job                                                                                                                                           | `jobs.cancel_job`       | `POST` [`/jobs/<job-id>/cancel`](/api-reference/api/job/cancel-job)                |

A workflow job is created automatically whenever:

* A workflow runs on a schedule. For more details see [Create workflow](/api-reference/api/workflow/create-workflow).
* You run a workflow manually. For more details see [Run workflow](/api-reference/api/workflow/run-workflow).


# Migration
Source: https://docs.unstructured.io/api-reference/workflow/migration



This page contains information about how to migrate from previous versions of the
Unstructured API's [workflow operations](/api-reference/workflow/overview).

## ConnectorConfigInput and ConnectorType classes are deprecated

**Applies to**: The [Unstructured Python SDK](/api-reference/workflow/overview#unstructured-python-sdk) only.

**Issue**: Referencing classes that end in `ConnectorConfigInput` such as `S3SourceConnectorConfigInput`
and `S3DestinationConnectorConfigInput`, and referencing classes that end in `ConnectorType` such as
`S3SourceConnectorType` and `S3DestinationConnectorType`, produce warnings at design time and errors at run time.

**Cause**: The preceding classes have been deprecated and are no longer supported.

**Solution**:

* Remove references to classes that end in `ConnectorConfigInput` from your code. Replace these references with a
  dictionary. This dictionary must contain the same fields as the
  class that you removed, with dictionary key/value pairs instead of object parameter/argument pairs.
* Remove references to classes that end in `ConnectorType` from your code. Replace these references with a string reference instead. This
  string corresponds to Unstructured's programmatic identifier for the connector type, for example `s3` for Amazon S3.

For example, the following code template shows the deprecated approach for programmatically creating an S3 source connector.

```python Python SDK theme={null}
# Deprecated code example. Do not use.
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
# The SourceConnectorType and S3SourceConnectorConfigInput classes are deprecated.
from unstructured_client.models.shared import (
    CreateSourceConnector,
    SourceConnectorType,         # <- Remove. Deprecated.
    S3SourceConnectorConfigInput # <- Remove. Deprecated.
)

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type=SourceConnectorType.S3, # <- Replace SourceConnectorType.S3 with a string reference to "s3".
                # Replace S3SourceConnectorConfigInput with a typed dictionary.
                config=S3SourceConnectorConfigInput(
                    # For anonymous authentication:
                    anonymous=True,

                    # For AWS access key ID with AWS secret access key authentication:
                    key="<key>", 
                    secret="<secret>", 

                    # For AWS STS token authentication:
                    token="<token>", 
                    key="<key>", 
                    secret="<secret>", 

                    remote_url="<remote_url>",
                    endpoint_url="<endpoint-url>", 
                    recursive=<True|False>
                )
            )
        )
    )

    print(response.source_connector_information)
```

To address this issue, use the following code template instead, which removes the
`SourceConnectorType` and `S3SourceConnectorConfigInput` class references and replaces them
with the correct substitutions.

```python Python SDK theme={null}
import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import CreateSourceConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type="s3",
                config={
                    # For anonymous authentication:
                    "anonymous": True,

                    # For AWS access key ID with AWS secret access key authentication:
                    "key": "<key>", 
                    "secret": "<secret>", 

                    # For AWS STS token authentication:
                    "token": "<token>", 
                    "key": "<key>", 
                    "secret": "<secret>", 

                    "remote_url": "<remote_url>",
                    "endpoint_url": "<endpoint-url>", 
                    "recursive": <True|False>
                }
            )
        )
    )

    print(response.source_connector_information)
```


# Available models
Source: https://docs.unstructured.io/api-reference/workflow/models

The following models are available for use in Unstructured workflow nodes, organized by model functionality and provider.

## Partitioner (VLM)

| Provider       | Model ID                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-opus-4-6`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929`                                                                                                                                                                                                                                                                                         |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                                                                                                                                                                                                                                               |
| `bedrock`      | `us.amazon.nova-lite-v1:0`<br />`us.amazon.nova-pro-v1:0`<br />`us.anthropic.claude-3-haiku-20240307-v1:0`<br />`us.anthropic.claude-3-opus-20240229-v1:0`<br />`us.anthropic.claude-3-sonnet-20240229-v1:0`<br />`us.anthropic.claude-opus-4-5-20251101-v1:0`<br />`us.anthropic.claude-opus-4-6-v1`<br />`us.anthropic.claude-sonnet-4-20250514-v1:0`<br />`us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`<br />`gpt-5.2`                                                                                                                                                                                                                                                                                                                                                |
| `vertexai`     | `gemini-2.0-flash-001`<br />`gemini-2.5-flash`<br />`gemini-2.5-pro`                                                                                                                                                                                                                                                                                                                                        |

## Embedder

| Provider       | Model ID                                                                                                                                                                                                   |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `azure_openai` | `text-embedding-3-large`<br />`text-embedding-3-small`<br />`text-embedding-ada-002`                                                                                                                       |
| `bedrock`      | `amazon.titan-embed-text-v1`<br />`amazon.titan-embed-text-v2:0`<br />`cohere.embed-english-v3`<br />`cohere.embed-multilingual-v3`<br />`amazon.titan-embed-image-v1`                                     |
| `ibm`          | `sentence-transformers/all-minilm-l6-v2`<br />`ibm/granite-embedding-278m-multilingual`<br />`intfloat/multilingual-e5-large`<br />`ibm/slate-125m-english-rtrvr-v2`<br />`ibm/slate-30m-english-rtrvr-v2` |
| `togetherai`   | `togethercomputer/m2-bert-80M-32k-retrieval`                                                                                                                                                               |
| `voyageai`     | `voyage-3`<br />`voyage-3-large`<br />`voyage-3-lite`<br />`voyage-code-2`<br />`voyage-code-3`<br />`voyage-finance-2`<br />`voyage-law-2`<br />`voyage-multimodal-3`                                     |

## Extract (LLM)

| Provider       | Model ID                                                                                                                                                                                  |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929`                                                                                              |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                             |
| `bedrock`      | `us.anthropic.claude-opus-4-5-20251101-v1:0`<br />`us.anthropic.claude-opus-4-6-v1`<br />`us.anthropic.claude-sonnet-4-20250514-v1:0`<br />`us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                             |

## Enrichment nodes

### Generative OCR

| Provider       | Model ID                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-opus-4-6`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929`                                                                                                                                                                                                                                                                                         |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                                                                                                                                                                                                                                               |
| `bedrock`      | `us.amazon.nova-lite-v1:0`<br />`us.amazon.nova-pro-v1:0`<br />`us.anthropic.claude-3-haiku-20240307-v1:0`<br />`us.anthropic.claude-3-opus-20240229-v1:0`<br />`us.anthropic.claude-3-sonnet-20240229-v1:0`<br />`us.anthropic.claude-opus-4-5-20251101-v1:0`<br />`us.anthropic.claude-opus-4-6-v1`<br />`us.anthropic.claude-sonnet-4-20250514-v1:0`<br />`us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`<br />`gpt-5.2`                                                                                                                                                                                                                                                                                                                                                |

### Image Description

| Provider       | Model ID                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-opus-4-6`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929`                                                                                                                                                                                                                                                                                         |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                                                                                                                                                                                                                                               |
| `bedrock`      | `us.amazon.nova-lite-v1:0`<br />`us.amazon.nova-pro-v1:0`<br />`us.anthropic.claude-3-haiku-20240307-v1:0`<br />`us.anthropic.claude-3-opus-20240229-v1:0`<br />`us.anthropic.claude-3-sonnet-20240229-v1:0`<br />`us.anthropic.claude-opus-4-5-20251101-v1:0`<br />`us.anthropic.claude-opus-4-6-v1`<br />`us.anthropic.claude-sonnet-4-20250514-v1:0`<br />`us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`<br />`gpt-5.2`                                                                                                                                                                                                                                                                                                                                                |
| `vertexai`     | `gemini-2.0-flash-001`<br />`gemini-2.5-flash`<br />`gemini-2.5-pro`                                                                                                                                                                                                                                                                                                                                        |

### Named Entity Recognition

| Provider       | Model ID                                                                                                            |
| -------------- | ------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-opus-4-6`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929` |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                       |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`<br />`gpt-5.2`                                                        |

### Table Description

| Provider       | Model ID                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-opus-4-6`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929`                                                                                                                                                                                                                                                                                         |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                                                                                                                                                                                                                                               |
| `bedrock`      | `us.amazon.nova-lite-v1:0`<br />`us.amazon.nova-pro-v1:0`<br />`us.anthropic.claude-3-haiku-20240307-v1:0`<br />`us.anthropic.claude-3-opus-20240229-v1:0`<br />`us.anthropic.claude-3-sonnet-20240229-v1:0`<br />`us.anthropic.claude-opus-4-5-20251101-v1:0`<br />`us.anthropic.claude-opus-4-6-v1`<br />`us.anthropic.claude-sonnet-4-20250514-v1:0`<br />`us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`<br />`gpt-5.2`                                                                                                                                                                                                                                                                                                                                                |
| `vertexai`     | `gemini-2.0-flash-001`<br />`gemini-2.5-flash`<br />`gemini-2.5-pro`                                                                                                                                                                                                                                                                                                                                        |

### Table to HTML

| Provider       | Model ID                                                                                                                                                                                                                                                                                                                                                                                                    |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic`    | `claude-opus-4-5-20251101`<br />`claude-opus-4-6`<br />`claude-sonnet-4-20250514`<br />`claude-sonnet-4-5-20250929`                                                                                                                                                                                                                                                                                         |
| `azure_openai` | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`                                                                                                                                                                                                                                                                                                                                                               |
| `bedrock`      | `us.amazon.nova-lite-v1:0`<br />`us.amazon.nova-pro-v1:0`<br />`us.anthropic.claude-3-haiku-20240307-v1:0`<br />`us.anthropic.claude-3-opus-20240229-v1:0`<br />`us.anthropic.claude-3-sonnet-20240229-v1:0`<br />`us.anthropic.claude-opus-4-5-20251101-v1:0`<br />`us.anthropic.claude-opus-4-6-v1`<br />`us.anthropic.claude-sonnet-4-20250514-v1:0`<br />`us.anthropic.claude-sonnet-4-5-20250929-v1:0` |
| `openai`       | `gpt-4o`<br />`gpt-4o-mini`<br />`gpt-5-mini`<br />`gpt-5.2`                                                                                                                                                                                                                                                                                                                                                |


# Chunker node
Source: https://docs.unstructured.io/api-reference/workflow/nodes/chunker/chunker

Configure a Chunker node to split partitioned document elements into chunks sized for embedding models, improving retrieval precision in RAG applications.

[Learn about the available chunking strategies](/concepts/chunking).

| Node                                                                                     | Description                                                                                        |
| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| [Chunk by Character](/api-reference/workflow/nodes/chunker/chunker-chunk-by-character)   | Combines elements within hard and soft character limits, without using section or page boundaries. |
| [Chunk by Title](/api-reference/workflow/nodes/chunker/chunker-chunk-by-title)           | Preserves section boundaries by starting a new chunk at each Title element.                        |
| [Chunk by Page](/api-reference/workflow/nodes/chunker/chunker-chunk-by-page)             | Preserves page boundaries, ensuring no chunk spans more than one page.                             |
| [Chunk by Similarity](/api-reference/workflow/nodes/chunker/chunker-chunk-by-similarity) | Groups topically similar consecutive elements into chunks using an embedding model.                |


# Chunk by Character strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/chunker/chunker-chunk-by-character

The Chunk by Character strategy combines sequential elements to fill each chunk within hard and soft character limits, without using section or page boundaries.

*Type*: `chunk`

*Subtype*: `chunk_by_character`

## Settings

<ParamField type="string">
  If specified, overrides the default API URL used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="string">
  If specified, overrides the default API key used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="boolean">
  If `true`, the elements used to form a chunk appear in `.metadata.orig_elements` for that chunk. Default: `false`.
</ParamField>

<ParamField type="integer">
  Soft maximum length of a chunk in characters. Closes a section after reaching approximately this length. Default: none.
</ParamField>

<ParamField type="integer">
  Hard maximum number of characters in a chunk. Default: none.
</ParamField>

<ParamField type="integer">
  Number of trailing characters from the prior text-split chunk to prepend to each subsequent chunk formed by splitting an oversized element. Default: none.
</ParamField>

<ParamField type="boolean">
  If `true`, applies overlap to chunks formed by combining whole elements, not just oversized ones. Use with caution — this can introduce noise into otherwise clean semantic units. Default: `false`.
</ParamField>

<ParamField type="boolean">
  If `true`, each table is placed in its own dedicated chunk, separate from any surrounding
  text elements. If `false`, small tables may share a chunk with adjacent text, producing
  mixed chunks that contain both text and table content.

  Regardless of this setting, narrative overlap is never prepended to table-only chunks,
  and table overlap is never appended to following narrative chunks.

  Note that `max_characters` applies to a chunk's visible `.text` content. When a table
  joins a mixed chunk with `isolate_table` set to `false`, the table's
  `metadata.text_as_html` is not subject to the same size budget. If the table has a large
  HTML representation, the serialized chunk payload may exceed `max_characters`. Use
  `isolate_table: true` if you have strict payload size requirements.

  Default: `true`.
</ParamField>

<ParamField type="string">
  If specified, prepends chunk-specific explanatory context to each chunk. Allowed value: `v1`. Default: none.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  chunk_by_character_chunker_workflow_node = WorkflowNode(
      name="Chunker",
      subtype="chunk_by_character",
      type="chunk",
      settings={
          "unstructured_api_url": None,
          "unstructured_api_key": None,
          "include_orig_elements": <True|False>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <True|False>,
          "isolate_table": <True|False>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Chunker",
      "type": "chunk",
      "subtype": "chunk_by_character",
      "settings": {
          "unstructured_api_url": null,
          "unstructured_api_key": null,
          "include_orig_elements": <true|false>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <true|false>,
          "isolate_table": <true|false>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>"
      }
  }
  ```
</RequestExample>


# Chunk by Page strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/chunker/chunker-chunk-by-page

The Chunk by Page strategy preserves page boundaries, ensuring no chunk contains content from more than one page.

*Type*: `chunk`

*Subtype*: `chunk_by_page`

## Settings

<ParamField type="string">
  If specified, overrides the default API URL used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="string">
  If specified, overrides the default API key used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="boolean">
  If `true`, the elements used to form a chunk appear in `.metadata.orig_elements` for that chunk. Default: `false`.
</ParamField>

<ParamField type="integer">
  Soft maximum length of a chunk in characters. Closes a section after reaching approximately this length. Default: none.
</ParamField>

<ParamField type="integer">
  Hard maximum number of characters in a chunk. Default: none.
</ParamField>

<ParamField type="integer">
  Number of trailing characters from the prior text-split chunk to prepend to each subsequent chunk formed by splitting an oversized element. Default: none.
</ParamField>

<ParamField type="boolean">
  If `true`, applies overlap to chunks formed by combining whole elements, not just oversized ones. Use with caution — this can introduce noise into otherwise clean semantic units. Default: `false`.
</ParamField>

<ParamField type="string">
  If specified, prepends chunk-specific explanatory context to each chunk. Allowed value: `v1`. Default: none.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  chunk_by_page_chunker_workflow_node = WorkflowNode(
      name="Chunker",
      subtype="chunk_by_page",
      type="chunk",
      settings={
          "unstructured_api_url": None,
          "unstructured_api_key": None,
          "include_orig_elements": <True|False>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <True|False>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Chunker",
      "type": "chunk",
      "subtype": "chunk_by_page",
      "settings": {
          "unstructured_api_url": null,
          "unstructured_api_key": null,
          "include_orig_elements": <true|false>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <true|false>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>"
      }
  }
  ```
</RequestExample>


# Chunk by Similarity strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/chunker/chunker-chunk-by-similarity

The Chunk by Similarity strategy uses an embedding model to identify topically similar consecutive elements and combines them into chunks.

*Type*: `chunk`

*Subtype*: `chunk_by_similarity`

## Settings

<ParamField type="string">
  If specified, overrides the default API URL used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="string">
  If specified, overrides the default API key used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="boolean">
  If `true`, the elements used to form a chunk appear in `.metadata.orig_elements` for that chunk. Default: `false`.
</ParamField>

<ParamField type="integer">
  Soft maximum length of a chunk in characters. Closes a section after reaching approximately this length. Default: none.
</ParamField>

<ParamField type="integer">
  Hard maximum number of characters in a chunk. Default: none.
</ParamField>

<ParamField type="integer">
  Number of trailing characters from the prior text-split chunk to prepend to each subsequent chunk formed by splitting an oversized element. Default: none.
</ParamField>

<ParamField type="boolean">
  If `true`, applies overlap to chunks formed by combining whole elements, not just oversized ones. Use with caution — this can introduce noise into otherwise clean semantic units. Default: `false`.
</ParamField>

<ParamField type="string">
  If specified, prepends chunk-specific explanatory context to each chunk. Allowed value: `v1`. Default: none.
</ParamField>

<ParamField type="number">
  Minimum similarity score required for consecutive elements to be combined into the same chunk. Must be between `0.0` and `1.0` exclusive (i.e., `0.01` to `0.99`). Default: none.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  chunk_by_similarity_chunker_workflow_node = WorkflowNode(
      name="Chunker",
      subtype="chunk_by_similarity",
      type="chunk",
      settings={
          "unstructured_api_url": None,
          "unstructured_api_key": None,
          "include_orig_elements": <True|False>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <True|False>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>",
          "similarity_threshold": <similarity-threshold>
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Chunker",
      "type": "chunk",
      "subtype": "chunk_by_similarity",
      "settings": {
          "unstructured_api_url": null,
          "unstructured_api_key": null,
          "include_orig_elements": <true|false>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <true|false>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>",
          "similarity_threshold": <similarity-threshold>
      }
  }
  ```
</RequestExample>


# Chunk by Title strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/chunker/chunker-chunk-by-title

The Chunk by Title strategy preserves section boundaries by starting a new chunk whenever a Title element is encountered.

*Type*: `chunk`

*Subtype*: `chunk_by_title`

## Settings

<ParamField type="string">
  If specified, overrides the default API URL used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="string">
  If specified, overrides the default API key used for chunker calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="boolean">
  If `true`, sections can span multiple pages. Default: `false`.
</ParamField>

<ParamField type="integer">
  Combines consecutive elements from a section into a single chunk until the section reaches this length in characters. Default: none.
</ParamField>

<ParamField type="boolean">
  If `true`, the elements used to form a chunk appear in `.metadata.orig_elements` for that chunk. Default: `false`.
</ParamField>

<ParamField type="integer">
  Soft maximum length of a chunk in characters. Closes a section after reaching approximately this length. Default: none.
</ParamField>

<ParamField type="integer">
  Hard maximum number of characters in a chunk. Default: none.
</ParamField>

<ParamField type="integer">
  Number of trailing characters from the prior text-split chunk to prepend to each subsequent chunk formed by splitting an oversized element. Default: none.
</ParamField>

<ParamField type="boolean">
  If `true`, applies overlap to chunks formed by combining whole elements, not just oversized ones. Use with caution — this can introduce noise into otherwise clean semantic units. Default: `false`.
</ParamField>

<ParamField type="boolean">
  If `true`, each table is placed in its own dedicated chunk, separate from any surrounding
  text elements. If `false`, small tables may share a chunk with adjacent text, producing
  mixed chunks that contain both text and table content.

  Regardless of this setting, narrative overlap is never prepended to table-only chunks,
  and table overlap is never appended to following narrative chunks.

  Note that `max_characters` applies to a chunk's visible `.text` content. When a table
  joins a mixed chunk with `isolate_table` set to `false`, the table's
  `metadata.text_as_html` is not subject to the same size budget. If the table has a large
  HTML representation, the serialized chunk payload may exceed `max_characters`. Use
  `isolate_table: true` if you have strict payload size requirements.

  Default: `true`.
</ParamField>

<ParamField type="string">
  If specified, prepends chunk-specific explanatory context to each chunk. Allowed value: `v1`. Default: none.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  chunk_by_title_chunker_workflow_node = WorkflowNode(
      name="Chunker",
      subtype="chunk_by_title",
      type="chunk",
      settings={
          "unstructured_api_url": None,
          "unstructured_api_key": None,
          "multipage_sections": <True|False>,
          "combine_text_under_n_chars": <combine-text-under-n-chars>,
          "include_orig_elements": <True|False>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <True|False>,
          "isolate_table": <True|False>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Chunker",
      "type": "chunk",
      "subtype": "chunk_by_title",
      "settings": {
          "unstructured_api_url": null,
          "unstructured_api_key": null,
          "multipage_sections": <true|false>,
          "combine_text_under_n_chars": <combine-text-under-n-chars>,
          "include_orig_elements": <true|false>,
          "new_after_n_chars": <new-after-n-chars>,
          "max_characters": <max-characters>,
          "overlap": <overlap>,
          "overlap_all": <true|false>,
          "isolate_table": <True|False>,
          "contextual_chunking_strategy": "<contextual-chunking-strategy>"
      }
  }
  ```
</RequestExample>


# Embedder node
Source: https://docs.unstructured.io/api-reference/workflow/nodes/embedder/embedder

Configure an Embedder node to convert chunked document elements into vector embeddings using a supported provider and model, enabling similarity search in vector stores.

*Type*: `embed`

*Subtype*: see [Settings](#settings)

<Warning>
  If you add an embedder node, you must set the workflow's chunker node's `max_characters` setting to a value at or below Unstructured's recommended
  maximum chunk size for your specified embedding model. [Learn more](/concepts/embedding#chunk-sizing-and-embedding-models).
</Warning>

[Learn about the available embedding providers and models](/concepts/embedding).

## Settings

<ParamField type="string">
  Embedding provider. Set at the `WorkflowNode` level, outside of `settings`. Allowed values: `azure_openai`, `bedrock`, `ibm`, `togetherai`, `voyageai`.

  <Note>
    The following list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

    For Unstructured **Business** accounts, to get your current list of available models, contact your
    Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
    [support@unstructured.io](mailto:support@unstructured.io).
  </Note>
</ParamField>

<ParamField type="string">
  Embedding model to use. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  embedder_workflow_node = WorkflowNode(
      name="Embedder",
      subtype="<subtype>",
      type="embed",
      settings={
          "model_name": "<model-name>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Embedder",
      "type": "embed",
      "subtype": "<subtype>",
      "settings": {
          "model_name": "<model-name>"
      }
  }
  ```
</RequestExample>


# Enrichment node
Source: https://docs.unstructured.io/api-reference/workflow/nodes/enrichment/enrichment

Configure an Enrichment node to add AI-powered enhancements to partitioned document elements, including image descriptions, table summaries, HTML representations, named entity recognition, and generative OCR.

[Learn about the available enrichments](/concepts/enriching/overview).

<Warning>
  Unstructured can potentially generate image summary descriptions, table summary descriptions, table-to-HTML output, and generative OCR optimizations, only for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is needed (or allowed).

  Even with these configurations, Unstructured actually generates image summary descriptions, table summary descriptions, and table-to-HTML output only for files that contain images or tables and are also eligible
  for processing with the following partitioning strategies:

  * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
  * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

  Unstructured never generates image summary descriptions, table summary descriptions, or table-to-HTML output for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Fast** partitioning strategy.
  * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images or tables.

  Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
</Warning>

| Node                                                                                                               | Description                                                             |
| ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| [Image Description](/api-reference/workflow/nodes/enrichment/enrichment-image-description)                         | Generates text summaries of detected images using a VLM.                |
| [Table Description](/api-reference/workflow/nodes/enrichment/enrichment-table-description)                         | Generates text summaries of detected tables using a VLM.                |
| [Table to HTML](/api-reference/workflow/nodes/enrichment/enrichment-table-to-html)                                 | Converts detected tables to HTML markup using agentic AI or a VLM.      |
| [Named Entity Recognition (NER)](/api-reference/workflow/nodes/enrichment/enrichment-named-entity-recognition-ner) | Identifies entities, their types, and relationships within text.        |
| [Generative OCR](/api-reference/workflow/nodes/enrichment/enrichment-generative-ocr)                               | Improves the fidelity of initially partitioned text blocks using a VLM. |


# Generative OCR task
Source: https://docs.unstructured.io/api-reference/workflow/nodes/enrichment/enrichment-generative-ocr

The Generative OCR task uses a vision language model to improve the fidelity of text blocks that Unstructured initially processed during partitioning.

*Type*: `prompter`

*Subtype*: see [Settings](#settings)

<Warning>
  Unstructured can produce generative OCR optimizations for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and a generative OCR optimizations node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No generative OCR optimization node is needed (or allowed).

  Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
</Warning>

<Warning>
  Generative OCR does not process any text blocks by default. You must also explicitly specify which document element
  types containing text that you want generative OCR to process.

  To do this, in a workflow's [High Res](/api-reference/workflow/nodes/partitioner/partitioner-high-res) partitioner node,
  add each [document element type](/concepts/document-elements#element-type) that you want generative OCR to process to the
  `extract_image_block_types` list within the partitioner node's `settings` field definition.

  Generative OCR does not process the text of any `Image` or `Table` elements if they have already been processed by
  [image description](/api-reference/workflow/nodes/enrichment/enrichment-image-description) or [table description](/api-reference/workflow/nodes/enrichment/enrichment-table-description) enrichments, respectively.
</Warning>

## Settings

<ParamField type="string">
  Enrichment algorithm and provider. Set at the `WorkflowNode` level, outside of `settings`. Allowed values:

  * `anthropic_ocr`
  * `bedrock_ocr`
  * `openai_ocr`

  <Note>
    The preceding list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

    For Unstructured **Business** accounts, to get your current list of available values, contact your
    Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
    [support@unstructured.io](mailto:support@unstructured.io).
  </Note>
</ParamField>

<ParamField type="string">
  Provider that matches the prefix of `subtype`. Allowed values: `anthropic`, `bedrock`, `openai`.
</ParamField>

<ParamField type="string">
  Model to use for generative OCR. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  generative_ocr_enrichment_workflow_node = WorkflowNode(
      name="Enrichment",
      subtype="<subtype>",
      type="prompter",
      settings={
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Enrichment",
      "type": "prompter",
      "subtype": "<subtype>",
      "settings": {
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  }
  ```
</RequestExample>


# Image Description task
Source: https://docs.unstructured.io/api-reference/workflow/nodes/enrichment/enrichment-image-description

The Image Description task uses a vision language model to generate text-based summaries of detected images in partitioned documents.

*Type*: `prompter`

*Subtype*: see [Settings](#settings)

<Warning>
  Unstructured can potentially generate image summary descriptions only for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node is needed (or allowed).

  Even with these configurations, Unstructured actually generates image summary descriptions only for files that contain images and are also eligible
  for processing with the following partitioning strategies:

  * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
  * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

  Unstructured never generates image summary descriptions for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Fast** partitioning strategy.
  * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images.
</Warning>

## Settings

<ParamField type="string">
  Enrichment algorithm and provider. Set at the `WorkflowNode` level, outside of `settings`. Allowed values:

  * `anthropic_image_description`
  * `bedrock_image_description`
  * `openai_image_description`
  * `vertexai_image_description`

  <Note>
    The preceding list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

    For Unstructured **Business** accounts, to get your current list of available values, contact your
    Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
    [support@unstructured.io](mailto:support@unstructured.io).
  </Note>
</ParamField>

<ParamField type="string">
  Provider that matches the prefix of `subtype`. Allowed values: `anthropic`, `bedrock`, `openai`, `vertexai`.
</ParamField>

<ParamField type="string">
  Model to use for image description. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  image_description_enrichment_workflow_node = WorkflowNode(
      name="Enrichment",
      subtype="<subtype>",
      type="prompter",
      settings={
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Enrichment",
      "type": "prompter",
      "subtype": "<subtype>",
      "settings": {
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  }
  ```
</RequestExample>


# Named Entity Recognition (NER) task
Source: https://docs.unstructured.io/api-reference/workflow/nodes/enrichment/enrichment-named-entity-recognition-ner

The Named Entity Recognition (NER) task identifies entities and their types — such as people, organizations, and locations — and the relationships between them.

*Type*: `prompter`

*Subtype*: see [Settings](#settings)

## Settings

<ParamField type="string">
  Enrichment algorithm and provider. Set at the `WorkflowNode` level, outside of `settings`. Allowed values:

  * `anthropic_ner`
  * `openai_ner`

  <Note>
    The preceding list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

    For Unstructured **Business** accounts, to get your current list of available values, contact your
    Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
    [support@unstructured.io](mailto:support@unstructured.io).
  </Note>
</ParamField>

<ParamField type="string">
  Provider that matches the prefix of `subtype`. Allowed values: `anthropic`, `openai`.
</ParamField>

<ParamField type="string">
  Model to use for NER. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<ParamField type="string">
  Alternative prompt to pass to the NER model. Default: none (uses Unstructured's internal default prompt).

  If specified, you must provide the complete prompt in the following format. For best results, limit changes to the entity types, relationship types, or clarifying instructions only — changing other parts of the prompt may produce unexpected results.

  ```text theme={null}
  Extract named entities and their relationships from the following text.

  Provide the entities, their corresponding types and relationships as a structured JSON response.

  Entity types:
  - PERSON
  - ORGANIZATION
  - LOCATION
  - DATE
  - TIME
  - EVENT
  - MONEY
  - PERCENT
  - FACILITY
  - PRODUCT
  - ROLE
  - DOCUMENT
  - DATASET

  Relationship types:
  - PERSON - ORGANIZATION: works_for, affiliated_with, founded
  - PERSON - LOCATION: born_in, lives_in, traveled_to
  - ORGANIZATION - LOCATION: based_in, has_office_in
  - Entity - DATE: occurred_on, founded_on, died_on, published_in
  - PERSON - PERSON: married_to, parent_of, colleague_of
  - PRODUCT - ORGANIZATION: developed_by, owned_by
  - EVENT - LOCATION: held_in, occurred_in
  - Entity - ROLE: has_title, acts_as, has_role
  - DATASET - PERSON: mentions
  - DATASET - DOCUMENT: located_in
  - PERSON - DATASET: published
  - DOCUMENT - DOCUMENT: referenced_in, contains
  - DOCUMENT - DATE: dated
  - PERSON - DOCUMENT: published

  [START OF TEXT]
  {{text}}
  [END OF TEXT]


  Response format json schema: {
      "items": [
          { "entity": "Entity name", "type": "Entity type" },
          { "entity": "Entity name", "type": "Entity type" }
      ],
      "relationships": [
          {"from": "Entity name", "relationship": "Relationship type", "to": "Entity name"},
          {"from": "Entity name", "relationship": "Relationship type", "to": "Entity name"}
      ]
  }
  ```
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  ner_enrichment_workflow_node = WorkflowNode(
      name="Enrichment",
      subtype="<subtype>",
      type="prompter",
      settings={
          "provider_type": "<provider-type>",
          "model": "<model>",
          "prompt_interface_overrides": {
              "prompt": {
                  "user": "<user-prompt-override>"
              }
          }
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Enrichment",
      "type": "prompter",
      "subtype": "<subtype>",
      "settings": {
          "provider_type": "<provider-type>",
          "model": "<model>",
          "prompt_interface_overrides": {
              "prompt": {
                  "user": "<user-prompt-override>"
              }
          }
      }
  }
  ```
</RequestExample>


# Table Description task
Source: https://docs.unstructured.io/api-reference/workflow/nodes/enrichment/enrichment-table-description

The Table Description task uses a vision language model to generate text-based summaries of detected tables in partitioned documents.

*Type*: `prompter`

*Subtype*: see [Settings](#settings)

<Warning>
  Unstructured can potentially generate table summary descriptions only for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and a table summary description node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No table summary description node is needed (or allowed).

  Even with these configurations, Unstructured actually generates table summary descriptions only for files that contain tables and are also eligible
  for processing with the following partitioning strategies:

  * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
  * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

  Unstructured never generates table summary descriptions for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Fast** partitioning strategy.
  * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain tables.
</Warning>

## Settings

<ParamField type="string">
  Enrichment algorithm and provider. Set at the `WorkflowNode` level, outside of `settings`. Allowed values:

  * `anthropic_table_description`
  * `bedrock_table_description`
  * `openai_table_description`
  * `vertexai_table_description`

  <Note>
    The preceding list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

    For Unstructured **Business** accounts, to get your current list of available values, contact your
    Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
    [support@unstructured.io](mailto:support@unstructured.io).
  </Note>
</ParamField>

<ParamField type="string">
  Provider that matches the prefix of `subtype`. Allowed values: `anthropic`, `bedrock`, `openai`, `vertexai`.
</ParamField>

<ParamField type="string">
  Model to use for table description. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  table_description_enrichment_workflow_node = WorkflowNode(
      name="Enrichment",
      subtype="<subtype>",
      type="prompter",
      settings={
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Enrichment",
      "type": "prompter",
      "subtype": "<subtype>",
      "settings": {
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  }
  ```
</RequestExample>


# Table to HTML task
Source: https://docs.unstructured.io/api-reference/workflow/nodes/enrichment/enrichment-table-to-html

The Table to HTML task converts detected tables into HTML markup using agentic AI or a vision language model, improving accuracy and downstream rendering.

*Type*: `prompter`

*Subtype*: see [Settings](#settings)

<Warning>
  Unstructured can potentially generate table-to-HTML output only for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and a table-to-HTML output node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No table-to-HTML output node is needed (or allowed).

  Even with these configurations, Unstructured actually generates table-to-HTML output only for files that contain tables and are also eligible
  for processing with the following partitioning strategies:

  * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
  * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

  Unstructured never generates table-to-HTML output for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Fast** partitioning strategy.
  * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain tables.
</Warning>

## Settings

<ParamField type="string">
  Enrichment algorithm and provider. Set at the `WorkflowNode` level, outside of `settings`. Allowed values:

  * `twopass_table2html` — agentic AI table-to-HTML output. Do not specify `provider_type` or `model` when using this subtype.
  * `anthropic_table2html` — VLM table-to-HTML output using Anthropic.
  * `openai_table2html` — VLM table-to-HTML output using OpenAI.

  <Note>
    The preceding list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

    For Unstructured **Business** accounts, to get your current list of available values, contact your
    Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
    [support@unstructured.io](mailto:support@unstructured.io).
  </Note>
</ParamField>

<ParamField type="string">
  Provider that matches the prefix of `subtype`. Not used with `twopass_table2html`. Allowed values: `anthropic`, `openai`.
</ParamField>

<ParamField type="string">
  Model to use for table-to-HTML conversion. Not used with `twopass_table2html`. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  table_to_html_enrichment_workflow_node = WorkflowNode(
      name="Enrichment",
      subtype="<subtype>",
      type="prompter",
      settings={
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Enrichment",
      "type": "prompter",
      "subtype": "<subtype>",
      "settings": {
          "provider_type": "<provider-type>",
          "model": "<model>"
      }
  }
  ```
</RequestExample>


# Extract node
Source: https://docs.unstructured.io/api-reference/workflow/nodes/extract/extract

Configure an Extract node to automatically pull structured data from partitioned documents into a consistent JSON format based on a schema or guidance prompt you provide.

[Learn about the available extraction methods](/concepts/structured-data-extractor/data-extractor).

| Node                                                         | Description                                                                                           |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| [LLM](/api-reference/workflow/nodes/extract/extract-llm)     | Extracts structured data using a language model with a JSON schema or plain-language guidance.        |
| [Regex](/api-reference/workflow/nodes/extract/extract-regex) | Extracts structured data using named regular-expression patterns, without requiring a language model. |


# LLM
Source: https://docs.unstructured.io/api-reference/workflow/nodes/extract/extract-llm

The LLM method uses a language model to extract structured data fields from partitioned documents based on a JSON schema or plain-language extraction guidance.

*Type*: `structured_data_extractor`

*Subtype*: `llm`

The following 4-minute video provides an overview of the structured data extractor.

<iframe title="YouTube video player" />

## Settings

<ParamField type="object">
  You must specify **exactly one** of the following. Structured LLM extraction always runs against a schema: either you supply that schema directly (`json_schema`), or you supply plain-language instructions (`extraction_guidance`) and Unstructured derives the extraction schema from that text first.
</ParamField>

<ParamField type="string">
  The extraction schema, in [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) format, for the structured data that you want to extract, expressed as a single string. Per OpenAI's guidelines, the maximum supported JSON schema nesting depth is 10 levels.
</ParamField>

<ParamField type="string">
  Plain-language instructions describing what to extract, expressed as a single string. Unstructured derives an extraction schema from this text (OpenAI Structured Outputs format) before the LLM performs structured extraction.
</ParamField>

<ParamField type="string">
  The mode in which to output the extracted data. Allowed values:

  * `elements_with_extracted_data`: Output the extracted data as JSON into an `extracted_data` field inside of `metadata` within a parent `DocumentData` element, followed by other built-in Unstructured document elements.
  * `extracted_data_only`: Output only the extracted data as JSON, without any parent `DocumentData` element or any other built-in Unstructured document elements.

  Default: `elements_with_extracted_data`.
</ParamField>

<ParamField type="string">
  LLM provider to use. Allowed values: `anthropic`, `azure_openai`, `bedrock`, `openai`.
</ParamField>

<ParamField type="string">
  LLM model to use. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

[Learn more](/concepts/structured-data-extractor/data-extractor).

<RequestExample>
  ```python Python SDK theme={null}
  extract_workflow_node = WorkflowNode(
      name="Extractor",
      subtype="llm",
      type="structured_data_extractor",
      settings={
          "schema_to_extract": {
              "json_schema": "<json-schema>",
              "extraction_guidance": "<extraction-guidance>"
          },
          "output_mode": "<elements_with_extracted_data | extracted_data_only>",
          "provider": "<provider>",
          "model": "<model>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Extractor",
      "type": "structured_data_extractor",
      "subtype": "llm",
      "settings": {
          "schema_to_extract": {
              "json_schema": "<json-schema>",
              "extraction_guidance": "<extraction-guidance>"
          },
          "output_mode": "<elements_with_extracted_data | extracted_data_only>",
          "provider": "<provider>",
          "model": "<model>"
      }
  }
  ```
</RequestExample>


# Regex
Source: https://docs.unstructured.io/api-reference/workflow/nodes/extract/extract-regex

The Regex method extracts structured data fields from partitioned documents using named regular-expression patterns, without requiring a language model.

*Type*: `structured_data_extractor`

*Subtype*: `regex`

## Settings

<ParamField type="array">
  A list of objects, each defining one extraction field. Each object must contain:

  * `name`: The field name for the extracted value.
  * `pattern`: A JSON-escaped regex pattern. Pattern syntax follows Python [`re`](https://docs.python.org/3/library/re.html).
</ParamField>

<ParamField type="string">
  The mode in which to output the extracted data. Allowed values:

  * `elements_with_extracted_data`: Output the extracted data as JSON into an `extracted_data` field inside of `metadata` within a parent `DocumentData` element, followed by other built-in Unstructured document elements.
  * `extracted_data_only`: Output only the extracted data as JSON, without any parent `DocumentData` element or any other built-in Unstructured document elements.

  Default: `elements_with_extracted_data`.
</ParamField>

## Regex output

Each of the `name` / `pattern` pairs in `regex_patterns` is extracted as an array of the matched values. The `name` field value is the name of the array in the extracted output.

See **Example settings** and **Example extracted output** below.

**Example settings:**

```json theme={null}
{
  "name": "Regex Extractor",
  "type": "structured_data_extractor",
  "subtype": "regex",
  "settings": {
    "regex_patterns": [
      {
        "name": "Price in dollars",
        "pattern": "\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?"
      }
    ],
    "output_mode": "extracted_data_only"
  }
}
```

**Note:** The `output_mode` field appears in the example settings above so the `settings` object is shown in full. It does not affect the extracted values illustrated in this example of the `Price in dollars` matches.

The extracted output includes the array `Price in dollars`.
The resulting array contains the values in the input document that match the regex `pattern`.

**Example extracted output (partial):**

```json theme={null}
{
  "Price in dollars": [
    "$249",
    "$498",
    "$189",
    "$567",
    "$59",
    "$118",
    "$1,183",
    "$224.77",
    "$1,407.77",
    "$25"
  ],
  ...
}
```

This example displays the output array for the following PDF file ([download this file](https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/pdf/invoice.pdf)).

The **example settings** above contain a JSON-escaped regex pattern.

| Context     | Pattern                                   |
| ----------- | ----------------------------------------- |
| Raw regex   | `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`      |
| JSON string | `\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?` |

<Tip>
  You can verify the list of matches for the regex `pattern` with a tool like [regex101](https://regex101.com/), with the flavor set to . Enter the raw regex string: `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`
</Tip>

![Sample invoice](https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/img/invoice.png)

## Regex pattern examples

The following are raw regex examples, shown without JSON escaping (as you would type them in a regex tester, not inside a JSON string value).

* Dollar amounts: `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`
  * This pattern matches dollar amounts with a dollar sign immediately followed by digits (no space after `$`), comma-separated thousands, and an optional decimal part.
  * Examples that match: `$2,345.99`, `$1,000,000`
  * Examples that don't match: `$ 2,345.99` (space follows the `$`), `1,000,000` (no `$`)
* Another string format: `^[A-Z]{2}-\d{2}-\d{4}$` - This pattern matches a string with two capital letters followed by two digits and a four-digit number, separated by dashes (example: `XY-12-1234`).
  * The `^` and `$` characters in this pattern are *anchors*. Use them to require that the entire string matches the pattern.
  * Do not use anchors if you need to extract values inline in text (like `Invoice id XY-12-1234`, `Patient record AB-34-5678`, `ID: AA-00-0000`)
  * Use anchors when the whole string must match the pattern exactly, with no extra text before or after (for example, `XY-12-1234` alone).
  * Examples that match: `XY-12-1234`, `AA-00-0000`
  * Examples that don't match: `Invoice XY-12-1234`, `XY-12-1234 ` (trailing space), `xy-12-1234`

<Tip>You can test and refine patterns by using tools such as [regex101](https://regex101.com/) with the flavor set to .</Tip>

## Regex escaping and JSON escaping

When using regex patterns in API requests, you may need two types of escaping:

* **Regex escaping** – required by the regex engine
* **JSON escaping** – required by the JSON string format

**Raw and escaped JSON examples:**

| Context     | Pattern                                   |
| ----------- | ----------------------------------------- |
| Raw regex   | `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`      |
| JSON string | `\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?` |
| Raw regex   | `^[A-Z]{2}-\d{2}-\d{4}$`                  |
| JSON string | `^[A-Z]{2}-\\d{2}-\\d{4}$`                |

**When to use each:**

* Use **regex escaping** when writing the pattern. This is considered *raw* regex.
  * `\$` → literal dollar sign
  * `\.` → literal decimal point

* Use **JSON escaping** when sending the pattern in an API request. You would add the JSON escapes to the raw regex, which may contain escaped literals like the examples above.
  * Each `\` becomes `\\`

From the visual schema editor on the **Start** page or in the **workflow editor**, use **Export schema as JSON** (from the three-dots menu) to download the schema with patterns **JSON-escaped** in the file. Or use **Upload JSON** to load a schema file so patterns show **raw** in the editor again.

<Tip>
  Write and test your regex first, then convert it to JSON by doubling each backslash. You can test and refine patterns by using tools such as [regex101](https://regex101.com/) with the flavor set to .
</Tip>

[Learn more](/concepts/structured-data-extractor/data-extractor).

<RequestExample>
  ```python Python SDK theme={null}
  extract_workflow_node = WorkflowNode(
      name="Regex Extractor",
      type="structured_data_extractor",
      subtype="regex",
      settings={
          "regex_patterns": [
              {
                  "name": "<field_name>",
                  "pattern": "<JSON escaped regex pattern>"
              },
              {
                  "name": "<field_name>",
                  "pattern": "<JSON escaped regex pattern>"
              }
          ],
          "output_mode": "<elements_with_extracted_data | extracted_data_only>"
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Regex Extractor",
      "type": "structured_data_extractor",
      "subtype": "regex",
      "settings": {
          "regex_patterns": [
              {
                  "name": "<field_name>",
                  "pattern": "<JSON escaped regex pattern>"
              },
              {
                  "name": "<field_name>",
                  "pattern": "<JSON escaped regex pattern>"
              }
          ],
          "output_mode": "<elements_with_extracted_data | extracted_data_only>"
      }
  }
  ```
</RequestExample>


# Custom workflow DAG nodes
Source: https://docs.unstructured.io/api-reference/workflow/nodes/overview



If `WorkflowType` is set to `CUSTOM` (for the Python SDK), or if `workflow_type` is set to `custom` (for `curl` or Postman), you must also specify the settings for the workflow's
directed acyclic graph (DAG) nodes. These nodes' settings are specified in the `workflow_nodes` array.

* A **Source** node is automatically created when you specify the `source_id` value outside of the\
  `workflow_nodes` array.

* A **Destination** node is automatically created when you specify the `destination_id` value outside of the
  `workflow_nodes` array.

* You can specify [Partitioner](/api-reference/workflow/nodes/partitioner/partitioner), [Enrichment](/api-reference/workflow/nodes/enrichment/enrichment),
  [Chunker](/api-reference/workflow/nodes/chunker/chunker), and [Embedder](/api-reference/workflow/nodes/embedder/embedder) nodes.

  <Warning>
    Unstructured can potentially generate image summary descriptions, table summary descriptions, table-to-HTML output, and generative OCR optimizations, only for workflows that are configured as follows:

    * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is added.
    * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is needed (or allowed).

    Even with these configurations, Unstructured actually generates image summary descriptions, table summary descriptions, and table-to-HTML output only for files that contain images or tables and are also eligible
    for processing with the following partitioning strategies:

    * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
    * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

    Unstructured never generates image summary descriptions, table summary descriptions, or table-to-HTML output for workflows that are configured as follows:

    * With a **Partitioner** node set to use the **Fast** partitioning strategy.
    * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images or tables.

    Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
  </Warning>

* The order of the nodes in the `workflow_nodes` array will be the same order that these nodes appear in the DAG,
  with the first node in the array added directly after the **Source** node. The **Destination** node
  follows the last node in the array.

* Be sure to specify nodes in the allowed order. The following DAG placements are all allowed:

```mermaid theme={null}
flowchart LR
    Partitioner
```

```mermaid theme={null}
flowchart LR
    Partitioner-->Chunker
```

```mermaid theme={null}
flowchart LR
    Partitioner-->Chunker-->Embedder
```

```mermaid theme={null}
flowchart LR
    Partitioner-->Enrichment-->Chunker
```

```mermaid theme={null}
flowchart LR
    Partitioner-->Enrichment-->Chunker-->Embedder
```

<Note>
  For workflows that use **Chunker** and enrichment nodes together, the **Chunker** node should be placed after all enrichment nodes. Placing the
  **Chunker** node before any enrichment nodes could cause incomplete or no enrichment results to be generated.
</Note>


# Partitioner node
Source: https://docs.unstructured.io/api-reference/workflow/nodes/partitioner/partitioner

Configure a Partitioner node to extract content from raw files and output it as structured document elements. Choose from Auto, VLM, High Res, and Fast strategies to balance output quality, speed, and cost.

[Learn about the available partitioning strategies](/concepts/partitioning).

| Node                                                                       | Description                                                                      |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| [Auto](/api-reference/workflow/nodes/partitioner/partitioner-auto)         | Routes each page to the optimal strategy at runtime, balancing quality and cost. |
| [VLM](/api-reference/workflow/nodes/partitioner/partitioner-vlm)           | Delivers highest-quality results for PDFs, images, and scanned documents.        |
| [High Res](/api-reference/workflow/nodes/partitioner/partitioner-high-res) | Processes all supported file types with advanced image-to-text extraction.       |
| [Fast](/api-reference/workflow/nodes/partitioner/partitioner-fast)         | Extracts text from text-only documents at the highest speed and lowest cost.     |


# Auto strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/partitioner/partitioner-auto

The Auto strategy dynamically routes each page to the Fast, High Res, or VLM partitioning strategy at runtime, optimizing for the highest quality at the lowest cost.

*Type*: `partition`

*Subtype*: `vlm`

## Settings

<ParamField type="string">
  VLM provider to use if Auto routing selects the VLM strategy. Allowed values: `anthropic`, `auto`, `bedrock`, `openai`, `vertexai`. Default: `anthropic`.
</ParamField>

<ParamField type="string">
  If specified, overrides the default API key used for VLM provider calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="string">
  VLM model to use if Auto routing selects the VLM strategy. Default: `claude-sonnet-4-20250514`.

  For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<ParamField type="string">
  Format of the partitioner response. Allowed values: `text/html`, `application/json`. Default: `text/html`.
</ParamField>

<ParamField type="boolean">
  If `true`, applies Beautiful Soup's `prettify` method to any HTML generated by the VLM partitioner, adding indentation for readability. Default: `true`.
</ParamField>

<ParamField type="boolean">
  If `true`, assigns UUIDs to element IDs, guaranteeing uniqueness (useful when used as database primary keys). If `false`, assigns a SHA-256 hash of the element's text as its ID. Default: `true`.
</ParamField>

<ParamField type="boolean">
  Must be `true` to enable dynamic routing of pages to Fast, High Res, or VLM as needed for optimal performance and cost.
</ParamField>

<ParamField type="boolean">
  Must be `true` to allow routing of pages to Fast partitioning for better performance and cost savings.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  auto_partitioner_workflow_node = WorkflowNode(
      name="Partitioner",
      subtype="vlm",
      type="partition",
      settings={
          "provider": "<provider>",
          "provider_api_key": None,
          "model": "<model>",
          "output_format": "<output-format>",
          "format_html": <True|False>,
          "unique_element_ids": <True|False>,
          "is_dynamic": True,
          "allow_fast": True
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Partitioner",
      "type": "partition",
      "subtype": "vlm",
      "settings": {
          "provider": "<provider>",
          "provider_api_key": null,
          "model": "<model>",
          "output_format": "<output-format>",
          "format_html": <true|false>,
          "unique_element_ids": <true|false>,
          "is_dynamic": true,
          "allow_fast": true
      }
  }
  ```
</RequestExample>


# Fast strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/partitioner/partitioner-fast

The Fast strategy uses rule-based text extraction optimized for text-only documents, delivering the fastest processing at the lowest cost.

*Type*: `partition`

*Subtype*: `unstructured_api`

## Settings

<ParamField type="string">
  Partitioning strategy to use. Must be set to `fast`.
</ParamField>

<ParamField type="boolean">
  If `true`, includes page breaks in the output where supported by the file type. Default: `false`.
</ParamField>

<ParamField type="boolean">
  Applies only to the `hi_res` strategy and has no effect here. Default: `false`.
</ParamField>

<ParamField type="array">
  List of Unstructured element types to exclude from the output. Default: none. Allowed values:

  * `Address`
  * `EmailAddress`
  * `FigureCaption`
  * `Footer`
  * `Formula`
  * `Header`
  * `Image`
  * `ListItem`
  * `NarrativeText`
  * `PageBreak`
  * `Table`
  * `Title`
  * `UncategorizedText`
</ParamField>

<ParamField type="boolean">
  If `true`, retains XML tags in the output. If `false`, extracts only the text content from XML tags. Default: `false`.
</ParamField>

<ParamField type="string">
  Encoding method used to decode the text input. Default: `utf-8`.
</ParamField>

<ParamField type="array">
  Languages present in the input, for use in partitioning, OCR, or both. Multiple values indicate the text could be in any of the specified languages. Default: `['eng']`. [See the language codes list](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/common/lang.py).
</ParamField>

<ParamField type="array">
  Unstructured element types for which image blocks are extracted as Base64-encoded data and stored in `metadata` fields. Default: none. Allowed values:

  * `Abstract`
  * `BulletedText`
  * `Caption`
  * `CodeSnippet`
  * `CompositeElement`
  * `Figure`
  * `FigureCaption`
  * `Form`
  * `FormKeysValues`
  * `Formula`
  * `Header`
  * `Image`
  * `List`
  * `List-item`
  * `ListItem`
  * `NarrativeText`
  * `Paragraph`
  * `Picture`
  * `Table`
  * `Text`
  * `Threading`
  * `Title`
  * `UncategorizedText`
</ParamField>

<ParamField type="boolean">
  If `true`, any table elements extracted from a PDF include an additional `text_as_html` metadata field containing an HTML `<table>` representation. Default: `false`.
</ParamField>

<ParamField type="boolean">
  If `true`, each element extracted from a PDF includes position information relative to its page. Default: `false`.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  fast_partitioner_workflow_node = WorkflowNode(
      name="Partitioner",
      subtype="unstructured_api",
      type="partition",
      settings={
          "strategy": "fast",
          "include_page_breaks": <True|False>,
          "pdf_infer_table_structure": <True|False>,
          "exclude_elements": [
              "<element-name>",
              "<element-name>"
          ],
          "xml_keep_tags": <True|False>,
          "encoding": "<encoding>",
          "ocr_languages": [
              "<language>",
              "<language>"
          ],
          "extract_image_block_types": [
              "image",
              "table"
          ],
          "infer_table_structure": <True|False>,
          "coordinates": <True|False>
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Partitioner",
      "type": "partition",
      "subtype": "unstructured_api",
      "settings": {
          "strategy": "fast",
          "include_page_breaks": <true|false>,
          "pdf_infer_table_structure": <true|false>,
          "exclude_elements": [
              "<element-name>",
              "<element-name>"
          ],
          "xml_keep_tags": <true|false>,
          "encoding": "<encoding>",
          "ocr_languages": [
              "<language-code>",
              "<language-code>"
          ],
          "extract_image_block_types": [
              "image",
              "table"
          ],
          "infer_table_structure": <true|false>,
          "coordinates": <true|false>
      }
  }
  ```
</RequestExample>


# High Res strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/partitioner/partitioner-high-res

The High Res strategy uses advanced image-to-text processing for all supported file types and supports generation of bounding box coordinates for document elements.

*Type*: `partition`

*Subtype*: `unstructured_api`

## Settings

<ParamField type="string">
  Partitioning strategy to use. Must be set to `hi_res`.
</ParamField>

<ParamField type="boolean">
  If `true`, includes page breaks in the output where supported by the file type. Default: `false`.
</ParamField>

<ParamField type="boolean">
  If `true`, any `Table` elements extracted from a PDF include an additional `text_as_html` metadata field containing the table data as an HTML table. Default: `false`.
</ParamField>

<ParamField type="array">
  List of Unstructured element types to exclude from the output. Default: none. Allowed values:

  * `Address`
  * `EmailAddress`
  * `FigureCaption`
  * `Footer`
  * `Formula`
  * `Header`
  * `Image`
  * `ListItem`
  * `NarrativeText`
  * `PageBreak`
  * `Table`
  * `Title`
  * `UncategorizedText`
</ParamField>

<ParamField type="boolean">
  If `true`, retains XML tags in the output. If `false`, extracts only the text content from XML tags. Default: `false`.
</ParamField>

<ParamField type="string">
  Encoding method used to decode the text input. Default: `utf-8`.
</ParamField>

<ParamField type="array">
  Languages present in the input, for use in partitioning, OCR, or both. Multiple values indicate the text could be in any of the specified languages. Default: `['eng']`. [See the language codes list](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/common/lang.py).
</ParamField>

<ParamField type="array">
  Unstructured element types for which image blocks are extracted as Base64-encoded data and stored in `metadata` fields. Default: none. Allowed values:

  * `Abstract`
  * `BulletedText`
  * `Caption`
  * `CodeSnippet`
  * `CompositeElement`
  * `Figure`
  * `FigureCaption`
  * `Form`
  * `FormKeysValues`
  * `Formula`
  * `Header`
  * `Image`
  * `List`
  * `List-item`
  * `ListItem`
  * `NarrativeText`
  * `Paragraph`
  * `Picture`
  * `Table`
  * `Text`
  * `Threading`
  * `Title`
  * `UncategorizedText`
</ParamField>

<ParamField type="boolean">
  If `true`, any table elements extracted from a PDF include an additional `text_as_html` metadata field containing an HTML `<table>` representation. Default: `false`.
</ParamField>

<ParamField type="boolean">
  If `true`, each element extracted from a PDF includes position information relative to its page. Default: `false`.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  high_res_paritioner_workflow_node = WorkflowNode(
      name="Partitioner",
      subtype="unstructured_api",
      type="partition",
      settings={
          "strategy": "hi_res",
          "include_page_breaks": <True|False>,
          "pdf_infer_table_structure": <True|False>,
          "exclude_elements": [
              "<element-name>",
              "<element-name>"
          ],
          "xml_keep_tags": <True|False>,
          "encoding": "<encoding>",
          "ocr_languages": [
              "<language>",
              "<language>"
          ],
          "extract_image_block_types": [
              "image",
              "table"
          ],
          "infer_table_structure": <True|False>,
          "coordinates": <True|False>
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Partitioner",
      "type": "partition",
      "subtype": "unstructured_api",
      "settings": {
          "strategy": "hi_res",
          "include_page_breaks": <true|false>,
          "pdf_infer_table_structure": <true|false>,
          "exclude_elements": [
              "<element-name>",
              "<element-name>"
          ],
          "xml_keep_tags": <true|false>,
          "encoding": "<encoding>",
          "ocr_languages": [
              "<language>",
              "<language>"
          ],
          "extract_image_block_types": [
              "image",
              "table"
          ],
          "infer_table_structure": <true|false>,
          "coordinates": <true|false>
      }
  }
  ```
</RequestExample>


# VLM strategy
Source: https://docs.unstructured.io/api-reference/workflow/nodes/partitioner/partitioner-vlm

The VLM strategy uses a vision language model to deliver the highest-quality partitioning for image-heavy file types including PDFs, images, and scanned documents.

*Type*: `partition`

*Subtype*: `vlm`

## Settings

<ParamField type="string">
  VLM provider to use. Allowed values: `anthropic`, `auto`, `bedrock`, `openai`, `vertexai`. Default: `anthropic`.
</ParamField>

<ParamField type="string">
  If specified, overrides the default API key used for VLM provider calls. Default: none (uses Unstructured's internal default).
</ParamField>

<ParamField type="string">
  VLM model to use. Default: `claude-sonnet-4-20250514`.

  For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).
</ParamField>

<ParamField type="string">
  Format of the partitioner response. Allowed values: `text/html`, `application/json`. Default: `text/html`.
</ParamField>

<ParamField type="boolean">
  If `true`, applies Beautiful Soup's `prettify` method to the HTML generated by the VLM partitioner, adding indentation for readability. Default: `true`.
</ParamField>

<ParamField type="boolean">
  If `true`, assigns UUIDs to element IDs, guaranteeing uniqueness (useful when used as database primary keys). If `false`, assigns a SHA-256 hash of the element's text as its ID. Default: `true`.
</ParamField>

<ParamField type="boolean">
  Must be `false` to use the VLM strategy.
</ParamField>

<ParamField type="boolean">
  If `true`, allows pages to be routed to Fast partitioning as needed for better performance and cost savings. Default: `true`.
</ParamField>

<RequestExample>
  ```python Python SDK theme={null}
  vlm_partitioner_workflow_node = WorkflowNode(
      name="Partitioner",
      subtype="vlm",
      type="partition",
      settings={
          "provider": "<provider>",
          "provider_api_key": None,
          "model": "<model>",
          "output_format": "<output-format>",
          "format_html": <True|False>,
          "unique_element_ids": <True|False>,
          "is_dynamic": False,
          "allow_fast": <True|False>
      }
  )
  ```

  ```json cURL theme={null}
  {
      "name": "Partitioner",
      "type": "partition",
      "subtype": "vlm",
      "settings": {
          "provider": "<provider>",
          "provider_api_key": null,
          "model": "<model>",
          "output_format": "<output-format>",
          "format_html": <true|false>,
          "unique_element_ids": <true|false>,
          "is_dynamic": false,
          "allow_fast": <true|false>
      }
  }
  ```
</RequestExample>


# Overview
Source: https://docs.unstructured.io/api-reference/workflow/overview



<Tip>
  To start using the Unstructured API's workflow operations right away, skip ahead to the [quickstart](#quickstart) or
  [walkthrough](#walkthrough),
  which uses the Unstructured Python SDK from remotely hosted Google Colab notebooks. These notebooks require no local setup.
</Tip>

The [Unstructured Pipelines](/pipelines/overview) features a no-code user interface for transforming your unstructured data into data that is ready
for retrieval-augmented generation (RAG).

The workflow operations in the [Unstructured API](/api-reference/overview) enable a full range of partitioning, chunking, embedding, and
enrichment options for your files and data. They are designed to batch-process files and data in remote locations; send processed results to
various storage, databases, and vector stores; and use the latest and highest-performing models. These workflow operations have built-in logic
to deliver the highest quality results at the lowest cost.

This page provides an overview of the Unstructured API's workflow operations. These operations enable Unstructured Pipelines automation usage
scenarios as well as for documentation, reporting, and recovery needs.

## Getting started

Choose one of the following options to get started with the Unstructured API's workflow operations:

* Follow the [quickstart](#quickstart) or [walkthrough](#walkthrough), which use the Unstructured Python SDK from remotely hosted Google Colab notebooks. These notebooks require no local setup.
* Start using the [Unstructred Python SDK](#unstructured-python-sdk). This option requires you to set up a local Python virtual environment.
* Start using a local [REST](#rest-endpoints) client, such as `curl` or Postman.

## Quickstart

The following quickstart shows how to use the [Unstructured Python SDK](/api-reference/workflow/overview#unstructured-python-sdk)
to have Unstructured process local files by using the Unstructured API's *on-demand jobs* functionality. This functionality
is part of the Unstructured API's collection of [workflow operations](https://docs.unstructured.io/api-reference/workflow/overview).

This quickstart is available in two options:

* [Use a remote notebook](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_On_Demand_Jobs_Quickstart.ipynb) - This option uses a remotely hosted Google Colab notebook. There are no additional setup steps required.
* [Use your local machine](/api-reference/quickstart/overview) - This option requires you to install the Unstructured Python SDK on your local machine.

<Note>
  The on-demand jobs functionality is designed to work *only by processing local files*.

  To process files (and data) in remote file and blob storage, databases, and vector stores, you must use other
  workflow operations in the Unstructured API. To learn how, see for example the notebook
  [Dropbox-To-Pinecone Connector API Quickstart for Unstructured](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Dropbox_To_Pinecone_Connector_Quickstart.ipynb).
</Note>

## Walkthrough

[This walkthrough](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_On_Demand_Jobs_Walkthrough.ipynb) builds upon the [quickstart](#quickstart).
Like the quickstart, this walkthrough shows how to use the [Unstructured Python SDK](/api-reference/workflow/overview#unstructured-python-sdk)
to have Unstructured process local files by using the Unstructured API's *on-demand jobs* functionality. This walkthrough goes further by
also showing in depth how to use Unstructured's [chunking](/concepts/chunking), [enriching](/concepts/enriching/overview), and [embedding](/concepts/embedding) features.
This walkthrough is available as a remotely hosted Google Colab notebook. This notebook requires no local setup.

<Note>
  The on-demand jobs functionality is designed to work *only by processing local files*.

  To process files (and data) in remote file and blob storage, databases, and vector stores, you must use other
  workflow operations in the Unstructured API. To learn how, see for example the notebook
  [Dropbox-To-Pinecone Connector API Quickstart for Unstructured](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Dropbox_To_Pinecone_Connector_Quickstart.ipynb).
</Note>

## Unstructured Python SDK

Watch the following 4-minute video to learn how to use the Python SDK to call the Unstructured API's workflow operations to
create [connectors](#connectors) in Unstructured Pipelines.

<iframe title="YouTube video player" />

Watch the following 4-minute video to learn how to use the Python SDK to call the Unstructured API's workflow operations to
create [workflows](#workflows) and [jobs](#jobs) in Unstructured Pipelines.

<iframe title="YouTube video player" />

Open a related [notebook](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Dropbox_To_Pinecone_Connector_Quickstart.ipynb) that covers many of
the concepts that are shown in the preceding videos.

The [Unstructured Python SDK](https://github.com/Unstructured-IO/unstructured-python-client), beginning with version 0.30.6,
allows you to call the Unstructured API's workflow operations through standard Python code.

To install the Unstructured Python SDK, run the following command from within your Python virtual environment:

```bash theme={null}
pip install "unstructured-client>=0.30.6"
```

If you already have the Unstructured Python SDK installed, upgrade to at least version 0.30.6 by running the following command instead:

```bash theme={null}
pip install --upgrade "unstructured-client>=0.30.6"
```

The Unstructured Python SDK code examples, shown later on this page and on related pages, use the following environment variable, which you can set as follows:

```bash theme={null}
export UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
```

This environment variable enables you to more easily run the following Unstructured Python SDK examples and help prevent
you from storing scripts that contain sensitive API keys in public source code repositories.

To get your Unstructured API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

Calls made by the Unstructured Python SDK's `unstructured_client` functions for creating, listing, updating,
and deleting connectors, workflows, and jobs in Unstructured Pipelines all use the Unstructured API's workflow operations URL.
This URL was provided to you when your Unstructured account was created.
If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  The default URL for the Unstructured API's workflow operations is `https://platform.unstructuredapp.io/api/v1`.
  However, you should always use the URL that was provided to you when your Unstructured account was created.
</Note>

To specify an API URL in your code, set the `server_url` parameter in the `UnstructuredClient` constructor to the target API URL.

The Unstructured API's workflow operations enable you to work with [connectors](#connectors),
[workflows](#workflows), and [jobs](#jobs) in Unstructured Pipelines.

* A *source connector* ingests files or data into Unstructured from a source location.
* A *destination connector* sends the processed data from Unstructured to a destination location.
* A *workflow* defines how Unstructured will process the data.
* A *job* runs a workflow at a specific point in time.

For general information about these objects, see:

* [Connectors](/pipelines/connectors)
* [Workflows](/pipelines/workflows)
* [Jobs](/pipelines/jobs)

Skip ahead to start learning about how to use the Unstructured Python SDK to work with
[connectors](#connectors),
[workflows](#workflows), and [jobs](#jobs) programmatically.

## REST endpoints

The Unstructured API's workflow operations are callable from a set of Representational State Transfer (REST) endpoints, which you can call through standard REST-enabled
utilities, tools, programming languages, packages, and libraries. The examples, shown later on this page and on related pages, describe how to call the Unstructured API's workflow operations with
`curl` and Postman. You can adapt this information as needed for your preferred programming languages and libraries, for example by using the
`requests` library with Python.

<Tip>
  You can also use the [Unstructured API's workflow operations - Swagger UI](https://platform.unstructuredapp.io/docs) to call the REST endpoints
  that are available through the default Unstructured API's workflow operations URL: `https://platform.unstructuredapp.io`. To use the Swagger UI, you must provide your Unstructured API key with each call. To
  get this API key, see the [quickstart](#quickstart), earlier on this page.

  Note that you should always use the URL that was provided to you when your Unstructured account was created.
  If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Tip>

### curl and Postman

The following `curl` examples use the following environment variables, which you can set as follows:

```bash theme={null}
export UNSTRUCTURED_API_URL="https://platform.unstructuredapp.io/api/v1"
export UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
```

For the API URL, this URL was provided to you when your Unstructured account was created.
If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  The default URL for the Unstructured API's workflow operations is `https://platform.unstructuredapp.io/api/v1`.
  However, you should always use the URL that was provided to you when your Unstructured account was created.
</Note>

These environment variables enable you to more easily run the following `curl` examples and help prevent
you from storing scripts that contain sensitive URLs and API keys in public source code repositories.

To get your Unstructured API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

The following Postman examples use variables, which you can set as follows:

1. In Postman, on your workspace's sidebar, click **Environments**.

2. Click **Globals**.

3. Create two global variables with the following settings:

   * **Variable**: `UNSTRUCTURED_API_URL`
   * **Type**: `default`
   * **Initial value**: The Unstructured API's workflow operations URL that was provided to you when your Unstructured account was created.
   * **Current value**: The Unstructured API's workflow operations URL that was provided to you when your Unstructured account was created.

   <br />

   * **Variable**: `UNSTRUCTURED_API_KEY`
   * **Type**: `secret`
   * **Initial value**: `<your-unstructured-api-key>`
   * **Current value**: `<your-unstructured-api-key>`

4. Click **Save**.

These variables enable you to more easily run the following examples in Postman and help prevent you from storing
Postman collections that contain sensitive URLs and API keys in public source code repositories.

Unstructured offers a [Postman collection](https://learning.postman.com/docs/collections/collections-overview/) that you can import into Postman to make Unstructured API's workflow operations requests through a graphical user interface.

1. [Install Postman](https://learning.postman.com/docs/getting-started/installation/installation-and-updates/).

2. [Sign in to Postman](https://learning.postman.com/docs/getting-started/installation/postman-account/#signing-in-to-postman).

3. In your workspace, click **Import**.

   <img alt="Import a Postman collection" />

4. In the **Paste cURL, Raw text or URL** box, enter the following URL, and then press `Enter`:

   For all workflow operations:

   ```text theme={null}
   https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/postman/Unstructured-REST-API-Workflow-Endpoint.postman_collection.json
   ```

   For [on-demand job](#run-an-on-demand-job) related operations only:

   ```text theme={null}
   https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/postman/Unstructured-REST-API-On-Demand-Jobs.postman_collection.json
   ```

   <Note>
     Each on-demand job is limited to 10 files, and each file is limited to 50 MB in size.

     If you need to launch a series of on-demand jobs in rapid succession, you must wait at least one second between launch
     requests. Otherwise, you will receive a rate limit error.

     A maximum of 5 on-demand jobs can be running in your Unstructured account. If you launch a new on-demand job
     but 5 existing on-demand jobs are still running, the new on-demand job will remain in a scheduled state until one of the 5
     existing on-demand jobs is done running.
   </Note>

5. On the sidebar, click **Collections**.

6. Expand **Unstructured REST API - Workflow Operations**.

7. Select the request that you want to use.

8. As applicable, modify the URL as needed to specify any required resource IDs for the request.

9. On the **Headers** tab, next to `unstructured-api-key`, enter your Unstructured API key in the **Value** column.
   As applicable, add, remove, or modify any other required headers for the request.

10. As applicable, on the **Params** tab, add, remove, or modify any required parameters for the request.

11. As applicable, on the **Body** tab, add, remove, or modify the required request body for the request.

12. Click **Send**.

13. To save the response, in the response area, click the ellipses, and then click **Save response to file**.

To get your Unstructured API key, do the following:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

The Unstructured API's workflow operations enable you to work with [connectors](#connectors),
[workflows](#workflows), and [jobs](#jobs) in Unstructured Pipelines.

* A *source connector* ingests files or data into Unstructured from a source location.
* A *destination connector* sends the processed data from Unstructured to a destination location.
* A *workflow* defines how Unstructured will process the data.
* A *job* runs a workflow at a specific point in time.

For general information about these objects, see:

* [Connectors](/pipelines/connectors)
* [Workflows](/pipelines/workflows)
* [Jobs](/pipelines/jobs)

Skip ahead to start learning about how to use the REST endpoints to work with
[connectors](#connectors),
[workflows](#workflows), and [jobs](#jobs) programmatically.

## Restrictions

The following Unstructured SDKs, tools, and libraries do *not* work with the Unstructured API's workflow operations:

* The [Unstructured JavaScript/TypeScript SDK](/api-reference/legacy-api/partition/sdk-jsts)
* [Local single-file POST requests](/api-reference/legacy-api/partition/sdk-jsts) to the legacy Unstructured Partition Endpoint
* The [Unstructured open source Python library](/open-source/introduction/overview)
* The [Unstructured Ingest CLI](/open-source/ingestion/ingest-cli)
* The [Unstructured Ingest Python library](/open-source/ingestion/python-ingest)

The following Unstructured API URL is also *not* supported: `https://api.unstructuredapp.io/general/v0/general` (the default legacy Unstructured Partition Endpoint URL).


# Retries
Source: https://docs.unstructured.io/api-reference/workflow/retries



For the [Unstructured Python SDK](/api-reference/workflow/overview#unstructured-python-sdk),
the [Unstructured API's workflow operations](/api-reference/workflow/overview) allows for retries with the
`RetryConfig` class when initializing the client, through the client's `retry_config` parameter.
If a request to the Unstructured API's workflow operations fails, the client will retry the
request with an exponential backoff strategy up to a maximum interval of one minute. The
function keeps retrying until the total elapsed time exceeds `max_elapsed_time`[\*](#parameter-names),
which defaults to one hour. However, you can override these defaults, for example as follows:

```python Python SDK theme={null}
# ...
from unstructured_client.utils import RetryConfig, BackoffStrategy

client = UnstructuredClient(
    api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")
    retry_config=RetryConfig(
        strategy="backoff",
        retry_connection_errors=True,
        backoff=BackoffStrategy(
            initial_interval=500,     # 0.5 seconds: 500 milliseconds
            max_interval=60000,       # 1 minute: 60 seconds * 1000 milliseconds
            exponent=1.5,
            max_elapsed_time=900000,  # 15 minutes: 15 * 60 seconds * 1000 milliseconds
        ),
    )
)
```

Available `RetryConfig` settings include:

| Setting                            | Description                                                                       |
| ---------------------------------- | --------------------------------------------------------------------------------- |
| `strategy`                         | The strategy to use for retries. The only supported value is `backoff`.           |
| `retry_connection_errors`          | True to retry on connection errors.                                               |
| `BackoffStrategy.initial_interval` | After the first error, wait the specified number of milliseconds before retrying. |
| `BackoffStrategy.max_interval`     | The maximum wait time, specified in milliseconds between retries.                 |
| `BackoffStrategy.exponent`         | After each retry, increase the wait time exponentially by the specified factor.   |
| `BackoffStrategy.max_elapsed_time` | Stop retrying after the specified number of milliseconds.                         |


# Azure Blob Storage
Source: https://docs.unstructured.io/api-reference/workflow/sources/azure-blob-storage



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Azure Blob Storage.

## Requirements

You will need:

The following video shows how to fulfill the minimum set of Azure Storage account requirements:

<iframe title="YouTube video player" />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  * **Read** and **List** for reading from the container only.
  * **Write** and **List** for writing to the container only.
  * **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these requirements:

* An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

  <iframe title="YouTube video player" />

* An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

  <iframe title="YouTube video player" />

  <iframe title="YouTube video player" />

* The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

  Create an SAS token (recommended):

  <iframe title="YouTube video player" />

  Get an access key or connection string:

  <iframe title="YouTube video player" />

## Examples

To create an Azure Blob Storage source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="azure",
                  config={
                      "remote_url": "az://<container-name>/<path/to/file/or/folder>",
                      "recursive": <True|False>,

                      # For anonymous authentication, omit the following auth keys.
                      
                      # For SAS token authentication:
                      # "account_name": "<account-name>",
                      # "sas_token": "<sas-token>",

                      # For account key authentication:
                      # "account_name": "<account-name>",
                      # "account_key": "<account-key>",

                      # For connection string authentication:
                      # "connection_string": "<connection-string>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "azure",
      "config": {
          "remote_url": "az://<container-name>/<path/to/file/or/folder>",
          "recursive": <true|false>,
      
          # For anonymous authentication, do not set any of the 
          # following fields.

          # For SAS token authentication:
          "account_name": "<account-name>",
          "sas_token": "<sas-token>"

          # For account key authentication:
          "account_name": "<account-name>",
          "account_key": "<account-key>"

          # For connection string authentication:
          "connection_string": "<connection-string>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`. For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the Azure Storage remote URL would be `az://my-container/my-folder/`.
</ParamField>

<ParamField type="string">
  The Azure Storage account name. Required for SAS token authentication and account key authentication.
</ParamField>

<ParamField type="string">
  For SAS token authentication, the SAS token for the Azure Storage account.
</ParamField>

<ParamField type="string">
  For account key authentication, the key for the Azure Storage account.
</ParamField>

<ParamField type="string">
  For connection string authentication, the connection string for the Azure Storage account.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to recursively access files from subfolders within the container.
</ParamField>

<h2>
  Set up Enterprise Connect authentication
</h2>

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see [Enterprise Connect for Azure](/business/azure/enterprise-connect).

To configure an Azure Blob Storage connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

1. Create an App Registration for Unstructured in Microsoft Entra ID.

   In your Azure subscription, follow the instructions in [How to register an app in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, `unstructured-connector`). For **Supported account types**, select **Single tenant only**.

   You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the [single-tenant scenario](https://learn.microsoft.com/en-us/entra/identity-platform/single-and-multi-tenant-apps) as defined by Microsoft.

2. Add a federated identity credential to the App Registration.

   Follow the instructions in [Configure an app to trust an external identity provider](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust) in the Microsoft Entra documentation. Navigate to your App Registration, select **Certificates & secrets** in the left navigation pane, select the **Federated credentials** tab, and select **Add credential**.

   For **Federated credential scenario**, select **Other issuer**.

   Set the following values:

   | Field        | Value                                                                                                                                                                                                                                                                                                            |
   | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**     | A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.                                                                                                                                                                                     |
   | **Issuer**   | The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)                                                                                                                                                       |
   | **Subject**  | The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error. |
   | **Audience** | Set this to `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                                        |

   Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

3. Add a role assignment to grant your App Registration access to your Azure Blob Storage account.

   See [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) in the Azure documentation. Use the following values:

   * **Scope**: the Azure Blob Storage account that contains the data you want the connector to access.
   * **Role**: select **Storage Blob Data Reader** for a source, or **Storage Blob Data Contributor** for a destination.
   * **Members**: select **User, group, or service principal**, then search for and select the App Registration you created in Step 1.

   When you reach the **Review + assign** tab, click **Review + assign** to complete the assignment.

4. Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the **Overview** page of your App Registration in the [Microsoft Entra admin center](https://entra.microsoft.com).

   * The **Tenant ID** (also called Directory ID) for your Azure subscription.
   * The **Client ID** of your App Registration.

Next, see the **Create the connector with Enterprise Connect** section below for examples.

### Create the source connector with Enterprise Connect

The following examples create an Azure Blob Storage source connector using Enterprise Connect authentication.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="azure",
                  config={
                      "remote_url": "az://<container-name>/<path/to/file/or/folder>",
                      "account_name": "<account-name>",
                      "tenant_id": "<tenant-id>",
                      "client_id": "<client-id>",
                      "recursive": <True|False>  # Boolean: True or False, no quotes
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "azure",
      "config": {
          "remote_url": "az://<container-name>/<path/to/file/or/folder>",
          "account_name": "<account-name>",
          "tenant_id": "<tenant-id>",
          "client_id": "<client-id>",
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`. For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the Azure Storage remote URL would be `az://my-container/my-folder/`.
</ParamField>

<ParamField type="string">
  The Azure Storage account name.
</ParamField>

<ParamField type="string">
  The Tenant ID (also called Directory ID) for your Azure subscription.
</ParamField>

<ParamField type="string">
  The Client ID of your App Registration.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to recursively access files from subfolders within the container.
</ParamField>


# Box
Source: https://docs.unstructured.io/api-reference/workflow/sources/box

Ingest your files into Unstructured from Box.

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

## Requirements

You will need:

<iframe title="YouTube video player" />

1. Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2. A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

   * In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
   * In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
   * Click **Save Changes** before continuing.

4. Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5. Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

   * In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
   * **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
   * Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6. The private key configuration JSON file for the Box Custom App, or a string that contains this file's contents.

   * To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location.
   * To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as
     a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself).
     To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
     In this command, replace `<path-to-downloaded-key-file>` with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.

     * For macOS or Linux:

       ```text theme={null}
       tr -d '\n' < <path-to-downloaded-key-file>
       ```

     * For Windows:

       ```text theme={null}
       (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
       ```

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

Permissions are defined in Box through *collaborations*, which function similar to access control lists. Collaboration objects are attached to folders, and those collaborations cascade down to everything inside that folder. A file's effective permissions come from the chain of ancestor folders above it. For more information, see [Collaborations overview](https://developer.box.com/guides/collaborations/) in the Box developer documentation.

To determine a file's effective permissions, Unstructured traverses the full folder hierarchy for each file, collecting collaborations at every level, and merging them. It also includes collaborations applied directly to the file itself. For more information, see [List folder collaborations](https://developer.box.com/reference/get-folders-id-collaborations/)  and [List file collaborations](https://developer.box.com/reference/get-files-id-collaborations/) in the Box developer documentation.

When Unstructured compiles a file's effective permissions:

* If the same user or group appears at multiple levels with different roles, Unstructured chooses the least-restrictive role.
* Unstructured does not include the following in the permission metadata. No sentinel value is written into the permission metadata.
  * `all_users_group`: Box's built-in group representing every user in the enterprise.
  * `is_access_only`: Collaboration property that determines whether, for collaborators, to display the items in the All Files list and let them see the path to the root folder for the shared item.
* Unstructured only includes collaborations with a status of `accepted`. It does not include collaborations with a status of `pending` or `rejected`.

Unstructured includes a maximum of 500 permissions per file.

Unstructured maps Box roles to `read`, `update`, and `delete` access:

| Box role             | Box access                                               | Unstructured permission metadata |
| -------------------- | -------------------------------------------------------- | -------------------------------- |
| `owner`              | Full access                                              | `read`, `update`, `delete`       |
| `co-owner`           | Full access except transfer ownership                    | `read`, `update`, `delete`       |
| `editor`             | View, download, upload, edit, delete, copy, move, rename | `read`, `update`                 |
| `viewer`             | Preview, download, comment                               | `read`                           |
| `previewer`          | Preview                                                  | `read`                           |
| `viewer uploader`    | Preview, download, upload                                | `read`                           |
| `previewer uploader` | Preview, upload                                          | `read`                           |
| `uploader`           | Upload and see item names                                | excluded entirely                |

Note the following:

* Unstructured excludes the `uploader` role because it confers no `read`, `update`, or `delete` access.
* Unstructured maps the Box `editor` role to `read` and `update` access but not `delete`, even though that role does allow deleting files.

For more information, see [Understanding collaborator permission level](https://support.box.com/hc/en-us/articles/360044196413-Understanding-Collaborator-Permission-Levels) at Box Support.

### Identifier format

Users and groups are identified by the numeric ID that Box assigns them. To retrieve information about a specific user or group, use appropriate Box API with the corresponding ID:

* [Get user](https://developer.box.com/reference/get-users-id/)
* [Get group](https://developer.box.com/reference/get-groups-id/)

### Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["11446688", "11336699"],
                            "groups": ["11223344"]
                        }
                    },
                    {
                        "update": {
                            "users": ["11446688"],
                            "groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

## Examples

To create an Box source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="box",
                  config={
                      "remote_url": "<remote-url>",
                      "recursive": <True|False>,
                      "box_app_config": "<box-app-config>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "box",
      "config": {
          "remote_url": "<remote-url>",
          "recursive": <true|false>,
          "box_app_config": "<box-app-config>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The URL to the target Box folder. This URL must take the format `box://<path/to/folder/in/account>`.
</ParamField>

<ParamField type="boolean">
  Set to `true` to recursively access files from subfolders within the target Box folder.
</ParamField>

<ParamField type="string">
  The contents of the private key configuration JSON file for the Box Custom App with access to the target Box folder. This file's contents must be provided as a single string.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Finding Needles in a Haystack: PII Detection at Scale with Unstructured, Box, and Elasticsearch](https://unstructured.io/blog/finding-needles-in-a-haystack-pii-detection-at-scale-with-unstructured-box-and-elasticsearch)


# Confluence
Source: https://docs.unstructured.io/api-reference/workflow/sources/confluence



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Confluence.

## Requirements

You will need:

* A [Confluence Cloud account](https://www.atlassian.com/software/confluence/pricing) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html).

* The site URL for your [Confluence Cloud account](https://community.atlassian.com/t5/Confluence-questions/confluence-cloud-url/qaq-p/1157148) or
  [Confluence Data Center installation](https://confluence.atlassian.com/confkb/how-to-find-your-site-url-to-set-up-the-confluence-data-center-and-server-mobile-app-938025792.html).

* A user in your [Confluence Cloud account](https://confluence.atlassian.com/cloud/invite-edit-and-remove-users-744721624.html) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html).

* The user must have the correct permissions in your
  [Conflunce Cloud account](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/permissions-and-restrictions-139557.html) to
  access the target spaces and pages.

* One of the following:

  * For Confluence Cloud or Confluence Data Center, the target user's name or email address, and password.
    [Change a Confluence Cloud user's password](https://support.atlassian.com/confluence-cloud/docs/change-your-confluence-password/).
    [Change a Confluence Data Center user's password](https://confluence.atlassian.com/doc/change-your-password-139416.html).
  * For Confluence Cloud only, the target user's name or email address, and API token.
    [Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
  * For Confluence Data Center only, the target user's personal access token (PAT).
    [Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).

* Optionally, the keys (not display names) of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access. To get a space's key,
  which is different from a space's display name, open the space in your web browser and look at the URL. The space's key is the part of the URL after `spaces/` but before the next `/` after that.

The following video provides related setup information for Confluence Cloud:

<iframe title="YouTube video player" />

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "11111:11111111-1111-1111-1111-111111111111"
                            ],
                            "groups": [
                                "22222222-2222-2222-2222-22222222",
                                "33333333-3333-3333-3333-33333333"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "44444:44444444-4444-4444-4444-44444444",
                                "55555:55555555-5555-5555-5555-55555555"
                            ],
                            "groups": [
                                "66666666-6666-6666-6666-66666666",
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "77777:77777777-7777-7777-7777-77777777"
                            ],
                            "groups": [
                                "88888888-8888-8888-8888-88888888"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Confluence user, use the user's ID (also known as their *account ID*) from the preceding output to call the
[GET /wiki/rest/api/user](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-users/#api-wiki-rest-api-user-get)
operation in the Confluence REST API.

To look up information about a particular Confluence group, use the group's ID from the preceding output to call the
[GET /wiki/rest/api/group/by-id](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-group/#api-wiki-rest-api-group-by-id-get)
operation in the Confluence REST API.

## Examples

To create a Confluence source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="confluence",
                  config={
                      "url": "<url>",
                      "max_num_of_spaces": <max-num-of-spaces>,
                      "max_num_of_docs_from_each_space": <max-num-of-docs-from-each-space>,
                      "spaces": ["<space-key>", "<space-key>"],
                      "extract_images": <True|False>,
                      "extract_files": <True|False>,

                      # For API token authentication:
                      # "username": "<username>",
                      # "token": "<api-token>",
                      # "cloud": <True|False>,

                      # For personal access token (PAT) authentication:
                      # "token": "<personal-access-token>",
                      # "cloud": False,

                      # For password authentication:
                      # "username": "<username>",
                      # "password": "<password>",
                      # "cloud": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "confluence",
      "config": {
          "url": "<url>",
          "max_num_of_spaces": <max-num-of-spaces>,
          "max_num_of_docs_from_each_space": <max-num-of-docs-from-each-space>,
          "spaces": ["<space-key>", "<space-key>"],
          "extract_images": "<true|false>",
          "extract_files": "<true|false>",

          # For API token authentication:

          "username": "<username>",
          "token": "<api-token>",
          "cloud": "<true|false>"

          # For personal access token (PAT) authentication:

          "token": "<personal-access-token>",
          "cloud": "false"

          # For password authentication:

          "username": "<username>",
          "password": "<password>",
          "cloud": "<true|false>"        
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The URL to the target Confluence Cloud instance.
</ParamField>

<ParamField type="integer">
  The maximum number of Confluence spaces to access within the Confluence Cloud instance.
</ParamField>

<ParamField type="integer">
  The maximum number of documents to access within each space.
</ParamField>

<ParamField type="string">
  An array of space keys (not display names) specifying the spaces to access, for example: `["luke","paul"]`. If no space keys are specified and `max_num_of_spaces` is exceeded for the instance, you might get unexpected results.
</ParamField>

<ParamField type="boolean">
  Set to `true` to download images and replace the HTML content with Base64-encoded images.
</ParamField>

<ParamField type="boolean">
  Set to `true` to download any embedded files in pages.
</ParamField>

<ParamField type="string">
  For username and API token authentication, or username and password authentication: the name or email address of the target user.
</ParamField>

<ParamField type="string">
  For username and API token authentication: the user's API token value. For personal access token (PAT) authentication: the target user's PAT value (set `cloud` to `false`).
</ParamField>

<ParamField type="boolean">
  For username and API token authentication, or username and password authentication: set to `true` if you are using Confluence Cloud.
</ParamField>

<ParamField type="string">
  For username and password authentication: the user's password.
</ParamField>


# Couchbase
Source: https://docs.unstructured.io/api-reference/workflow/sources/couchbase



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Couchbase.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Couchbase Capella clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe title="YouTube video player" />

For Couchbase Capella, you will need:

* A [Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier).
* A [Couchbase Capella cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started).
* A [bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket),
  [scope](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-scope),
  and [collection](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-collection)
  on the cluster.
* The cluster's [public connection string](https://docs.couchbase.com/cloud/get-started/connect.html#connect-from-sdk-cbsh-cli-or-ide).
* The [cluster access name (username) and secret (password)](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials).
* [Incoming IP address allowance](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) for the cluster.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

For a local Couchbase server, you will need:

* [Installation of a local Couchbase server](https://docs.couchbase.com/server/current/getting-started/start-here.html).
* [Connection details](https://docs.couchbase.com/server/current/guides/connect.html) to the local Couchbase server.

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

## Examples

To create a Couchbase source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="couchbase",
                  config={
                      "username": "<username>",
                      "bucket": "<bucket>",
                      "connection_string": "<connection-string>",
                      "scope": "<scope>",
                      "collection": "<collection>",
                      "password": "<password>",
                      "batch_size": <batch-size>,
                      "collection_id": "<collection-id>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "couchbase",
      "config": {
          "username": "<username>",
          "bucket": "<bucket>",
          "connection_string": "<connection-string>",
          "scope": "<scope>",
          "collection": "<collection>",
          "password": "<password>",
          "batch_size": <batch-size>,
          "collection_id": "<collection-id>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The username for the Couchbase server.
</ParamField>

<ParamField type="string">
  The name of the bucket in the Couchbase server.
</ParamField>

<ParamField type="string">
  The connection string for the Couchbase server.
</ParamField>

<ParamField type="string">
  The name of the scope in the bucket.
</ParamField>

<ParamField type="string">
  The name of the collection in the scope.
</ParamField>

<ParamField type="string">
  The password for the Couchbase server.
</ParamField>

<ParamField type="integer">
  The maximum number of records to transmit per batch.
</ParamField>

<ParamField type="string">
  Source connector only. The name of the collection field that contains the document ID.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Couchbase Integration in Unstructured Platform](https://unstructured.io/blog/couchbase-integration-in-unstructured-platform)


# Databricks Volumes
Source: https://docs.unstructured.io/api-reference/workflow/sources/databricks-volumes



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Databricks Volumes.

## Requirements

You will need:

* A Databricks account on [AWS](https://docs.databricks.com/getting-started/free-trial.html),
  [Azure](https://learn.microsoft.com/azure/databricks/getting-started/), or
  [GCP](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or
  [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* The workspace's URL. Get the workspace URL for
  [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
  [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
  or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

  Examples:

  * AWS: `https://<workspace-id>.cloud.databricks.com`
  * Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
  * GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

  <Note>
    Do not add a trailing slash (`/`) to the workspace URL.
  </Note>

* The Databricks authentication details. For more information, see the documentation for
  [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
  or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

  For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), the following Databricks authentication types are supported:

  * Databricks OAuth machine-to-machine (M2M) authentication for\
    [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

    You will need the **Client ID** (or **UUID** or **Application** ID) and OAuth **Secret** (client secret) values for the corresponding service principal.
    Note that for Azure, only Databricks managed service principals are supported. Microsoft Entra ID managed service principals are not supported.

    The following video shows how to create a Databricks managed service principal:

    <iframe title="YouTube video player" />

  * Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html).

    You will need the personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  For [Unstructured Ingest](/open-source/ingestion/overview), the following Databricks authentication types are supported:

  * For Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html): The personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  * For username and password (basic) authentication ([AWS](https://docs.databricks.com/archive/dev-tools/basic.html) only): The user's name and password values.

  * For OAuth machine-to-machine (M2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html)): The client ID and OAuth secret values for the corresponding service principal.

  * For OAuth user-to-machine (U2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-u2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-u2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-u2m.html)): No additional values.

  * For Azure managed identities (formerly Managed Service Identities (MSI) authentication) ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-mi) only): The client ID value for the corresponding managed identity.

  * For Microsoft Entra ID service principal authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-sp) only): The tenant ID, client ID, and client secret values for the corresponding service principal.

  * For Azure CLI authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-cli) only): No additional values.

  * For Microsoft Entra ID user authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/user-aad-token) only): The Entra ID token for the corresponding Entra ID user.

  * For Google Cloud Platform credentials authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-creds.html) only): The local path to the corresponding Google Cloud service account's credentials file.

  * For Google Cloud Platform ID authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-id.html) only): The Google Cloud service account's email address.

* The name of the parent catalog in Unity Catalog for
  [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
  [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
  [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html) for the volume.

* The name of the parent schema (formerly known as a database) in Unity Catalog for
  [AWS](https://docs.databricks.com/schemas/create-schema.html),
  [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
  [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) for the volume.

* The name of the volume in Unity Catalog for [AWS](https://docs.databricks.com/tables/managed.html),
  [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
  [GCP](https://docs.gcp.databricks.com/tables/managed.html), and optionally any path in that volume that you want to access directly, beginning with the volume's root.

* The Databricks workspace user or service principal must have the following *minimum* set of privileges to read from or write to the
  existing volume in Unity Catalog:

  * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
  * `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
  * `READ VOLUME` and `WRITE VOLUME` on the volume.

  The following videos show how to create and set privileges for a catalog, schema (formerly known as a database), and volume in Unity Catalog.

  <iframe title="YouTube video player" />

  Learn more about how to check and set Unity Catalog privileges for
  [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

## Examples

To create a Databricks Volumes source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="databricks_volumes",
                  config={
                      "catalog": "<catalog>",
                      "host": "<host>",
                      "schema_": "<schema>",
                      "volume": "<volume>",
                      "volume_path": "<volume_path>",

                      # For Databricks OAuth machine-to-machine (M2M) authentication:
                      # "client_id": "<client_id>",
                      # "client_secret": "<client_secret>",

                      # For Databricks personal access token authentication:
                      # "token": "<token>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "databricks_volumes",
      "config": {
          "host": "<host>",
          "catalog": "<catalog>",
          "schema": "<schema>",
          "volume": "<volume>",
          "volume_path": "<volume_path>",

          # For Databricks OAuth machine-to-machine (M2M) authentication:
          "client_id": "<client-id>"
          "client_secret": "<client-secret>"

          # For Databricks personal access token authentication:
          "token": "<token>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Databricks workspace host URL.

  <Note>
    Do not add a trailing slash (`/`) to the workspace host URL.
  </Note>
</ParamField>

<ParamField type="string">
  For Databricks OAuth machine-to-machine (M2M) authentication, the **Client ID** (or **UUID** or **Application ID**) value for the Databricks managed service principal that has the appropriate privileges to the volume.
</ParamField>

<ParamField type="string">
  For Databricks OAuth M2M authentication, the associated OAuth **Secret** value for the Databricks managed service principal that has the appropriate privileges to the volume.
</ParamField>

<ParamField type="string">
  For Databricks personal access token authentication, the personal access token's value.
</ParamField>

<ParamField type="string">
  The name of the catalog to use.
</ParamField>

<ParamField type="string">
  The name of the associated schema.
</ParamField>

<ParamField type="string">
  The name of the associated volume.
</ParamField>

<ParamField type="string">
  Any optional path to access within the volume.
</ParamField>


# Dropbox
Source: https://docs.unstructured.io/api-reference/workflow/sources/dropbox



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Dropbox.

## Requirements

You will need:

<iframe title="YouTube video player" />

1. A [Dropbox account](https://www.dropbox.com/try/teams).

2. A Dropbox app for your Dropbox account. To create a Dropbox app, do the following:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click **Create app**.<br />
   d) For **Choose an API**, select **Scoped access**.<br />
   e) For **Choose the type of access you need**, select **App folder**.<br />
   f) Enter a name for your app, and then click **Create app**.<br />
   g) On the app's **Permissions** tab, under **Files and folders**, check the boxes labelled **files.content.read** or **files.content.write** or both,
   depending on whether you want to read files, write files, or both. Then click **Submit**.<br />
   h) On the app's **Settings** tab, note the value of the **App folder name** field. This is the name of the app folder that Dropbox will create under the `Apps` top-level folder in your Dropbox
   account that the Dropbox app will use for access. If you change the value of **App folder name** field here, Dropbox will create an app folder with that name under the `Apps` top-level folder instead.<br />
   i) Under **OAuth 2**, next to **Generated access token**, click **Generate**. Copy the value of this access token. You should only click **Generate** after you have completed all of the preceding steps first.
   This is because the access token is scoped to the specific app folder and settings at the time the access token is generated. If you change the app folder name or any of the permissions later,
   you should regenerate the access token.<br />

   <Warning>
     Access tokens are valid for **only four hours** after they are created. After this four-hour period, you can no longer use the expired access token.
     Dropbox does not allow the creation of access tokens that are valid for more than four hours.

     To replace an expired access token, you must first generate a *refresh token* for the corresponding access token. To learn how to generate an access token and its corresponding refresh token,
     see [Replace an expired access token](#replace-an-expired-access-token), later in this article.

     If you do not already have the corresponding refresh token for an existing access token, or if you lose a refresh token after you generate it,
     you must generate a new access token and its corresponding refresh token.

     Instead of continualy replacing expired access tokens yourself, you can have Unstructured do it for you as needed; just supply Unstructured
     with the refresh token along with the Dropbox app's **App key** and **App secret** values.
     To learn how to supply these to Unstructured, look for mentions of "refresh token," "app key," and "app secret" in the connector settings later in this article.
   </Warning>

3. The app folder that your Dropbox app will use for access can be found in your Dropbox account under the `Apps` top-level folder. For example, if the value of the **App folder name**
   field above is `my-folder`, then the app folder that your Dropbox app will use for access can be found under `https://dropbox.com/home/Apps/my-folder`

   <img alt="The my-folder app folder under the Apps top-level folder" />

   <Warning>
     Your Dropbox app will *not* have access to upload or download files from the root of the app folder. Instead, you *must* create a subfolder inside of the app folder for your Dropbox
     app to upload or download files from. You will use the name of that subfolder when specifying your remote URL in the next step. For example, if your Dropbox app uses an app folder named `my-folder`
     for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then the subfolder that your Dropbox app will upload and download files from
     can be found under `https://dropbox.com/home/Apps/my-folder/data`

     <img alt="The data subfolder under the my-folder subfolder" />
   </Warning>

4. Note the remote URL to your subfolder inside of the app folder, which takes the format `dropbox://<subfolder-name>`. For example,
   if your Dropbox app uses an app folder named `my-folder` for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then
   the remote URL is `dropbox://data`

   <img alt="The data subfolder under the my-folder subfolder" />

## Replace an expired access token

Dropbox app access tokens are valid for **only four hours**. After this time, you can no longer use the expired access token.

To have Unstructured automatically replace expired access tokens on your behalf, do the following:

<iframe title="YouTube video player" />

1. Get the app key and app secret values for your Dropbox app. To do this:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click your Dropbox app's icon.<br />
   d) On the **Settings** tab, next to **App key**, copy the value of the app key.<br />
   e) Next to **App secret**, click **Show**, and then copy the value of the app secret.

2. Use your web browser to browse to the following URL, replacing `<app-key>` with the app key for your Dropbox app:

   ```text theme={null}
   https://www.dropbox.com/oauth2/authorize?client_id=<app-key>&response_type=code&token_access_type=offline
   ```

3. Click **Continue**.

4. Click **Allow**.

5. In the **Access code generated** tile, copy the access code that is shown.

6. Use the [curl](https://curl.se/) utility in your Terminal or Command Prompt, or use a REST API client such as
   [Postman](https://www.postman.com/product/api-client/), to make the following REST API call, replacing the following placeholders:

   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.
   * Replace `<access-code>` with the access code that you just copied.

   ```text theme={null}
   curl --location --request POST 'https://api.dropbox.com/oauth2/token' \
   --user '<app-key>:<app-secret>' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data-urlencode 'code=<access-code>' \
   --data-urlencode 'grant_type=authorization_code'
   ```

7. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token.
   * The value of `refresh_token` is the refresh token that can be used to replace this access token much faster and easier next time.
     If you lose this refresh token, you must go back to Step 2.

   For the [Unstructured Pipelines](/pipelines/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the connector settings **Refresh token** field.
   * Add the `<app-key>` value to the connector settings **App key** field.
   * Add the `<app-secret>` value to the connector settings **App secret** field.

   For the [Unstructured API](/api-reference/overview) and [Unstructured Ingest](/open-source/ingestion/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the `refresh_token` parameter.
   * Add the `<app-key>` value to the `app_key` parameter.
   * Add the `<app-secret>` value to the connector settings `app_secret` parameter.

8. If for some reason you need to manually replace the expired access token yourself instead of having Unstructured do it for you, you can use the refresh token that you just copied to get a new access token:

   * Replace `<refresh-token>` with the refresh token.
   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.

   ```text theme={null}
   curl https://api.dropbox.com/oauth2/token \
   --data refresh_token=<refresh-token> \
   --data grant_type=refresh_token \
   --data client_id=<app-key> \
   --data client_secret=<app-secret>
   ```

9. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token. In the connector, replace the old,
     expired access token value with this new, valid access token value.
   * The value of `refresh_token` is the new, valid refresh token. To replace the expired access token yourself, go back to Step 8.

## Examples

To create a Dropbox source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="dropbox",
                  config={
                      "remote_url": "<remote-url>",
                      "recursive": <True|False>,
                      "refresh_token": "<refresh-token>",
                      "app_key": "<app-key>",
                      "app_secret": "<app-secret>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "dropbox",
      "config": {
          "remote_url": "<remote-url>",
          "recursive": <true|false>,
          "refresh_token": "<refresh-token>",
          "app_key": "<app-key>",
          "app_secret": "<app-secret>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The remote URL to the target subfolder inside of the app folder for the Dropbox app.
</ParamField>

<ParamField type="boolean">
  Set to `true` to recursively process data from subfolders within the target subfolder.
</ParamField>

<ParamField type="string">
  The app key for your Dropbox app. This allows Unstructured to automatically replace expired access tokens.
</ParamField>

<ParamField type="string">
  The app secret for your Dropbox app. This allows Unstructured to automatically replace expired access tokens.
</ParamField>

<ParamField type="string">
  The refresh token for the Dropbox app. This allows Unstructured to automatically replace expired access tokens.
</ParamField>


# Elasticsearch
Source: https://docs.unstructured.io/api-reference/workflow/sources/elasticsearch



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Elasticsearch.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Elastic Cloud instances are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

* For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

  <iframe title="YouTube video player" />

* For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

  <iframe title="YouTube video player" />

* You will need the name of the index on the instance.

  For the destination connector, if you need to create an index, you can use for example the following `curl` command. Replace the following placeholders:

  * Replace `<host>:<port>` with the instance's host identifier and port number.

  * Replace one of the following, and remove the other:

    * For an instance's user and password combination, replace `<username>` with your Elasticsearch or Elastic Cloud instance's user name, replace `<password>` with your password, and then remove `--user "Authorization: ApiKey <your-api-key>"`.
    * For an Elastic Cloud API key, replace `<your-api-key>` with your Elastic Cloud API key, and then remove `--user "<username>:<password>"`.

  * Replace `<index-name>` with the name of the new index on the instance.

  * Replace `<index-schema>` with the schema for the index. A schema is optional; see the explanation
    following this `curl` command for more information.

  ```bash theme={null}
  curl --request PUT "<host>:<port>/<index-name>" \
  --user "<username>:<password>" \
  --user "Authorization: ApiKey <your-api-key>" \
  [--header "Content-Type: application/json" \
  --data '<index-schema>']
  ```

  To learn more, see [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

  For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
  Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
  to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  For objects in the `metadata` field that Unstructured produces and that you want to store in an Elasticsearch or Elastic Cloud index, you must create fields in your index's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    },
    "entities": {
      "items": [
        {
          "entity": "HOME FOR FUTURE",
          "type": "ORGANIZATION"
        },
        {
          "entity": "221 Queen Street, Melbourne VIC 3000",
          "type": "LOCATION"
        }
      ],
      "relationships": [
        {
          "from": "HOME FOR FUTURE",
          "relationship": "based_in",
          "to": "221 Queen Street, Melbourne VIC 3000"
        }
      ]
    }
  }
  ```

  You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
  required by Unstructured whenever you create your own index schema:

  * `element_id`
  * `record_id`, which is required by Unstructured for intelligent record updates.
  * `type`, which is not required, but highly recommended.
  * `text`
  * `embeddings` if embeddings are generated; make sure to set `dims` to the same number of dimensions as the embedding model generates.

  ```json theme={null}
  {
    "mappings": {
      "properties": {
        "element_id": {
          "type": "keyword"
        },
        "record_id": {
          "type": "text"
        },
        "text": {
          "type": "text"
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "embeddings": {
          "type": "dense_vector",
          "dims": 1536,
          "index": true,
          "similarity": "cosine"
        },
        "metadata": {
          "properties": {
            "is_extracted": {
              "type": "boolean"
            },
            "coordinates-points": {
              "type": "float"
            },
            "coordinates-system": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates-layout_width": {
              "type": "long"
            },
            "coordinates-layout_height": {
              "type": "long"
            },
            "filetype": {
              "type": "keyword"
            },
            "languages": {
              "type": "keyword"
            },
            "page_number": {
              "type": "integer"
            },
            "image_mime_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "data_source-url": {
              "type": "keyword"
            },
            "data_source-record_locator-protocol": {
              "type": "keyword"
            },
            "data_source-record_locator-remote_file_path": {
              "type": "keyword"
            },
            "entities-items": {
              "properties": {
                "entity": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "entities-relationships": {
              "properties": {
                "from": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "relationship": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "to": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
  * [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
  * [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  <iframe title="YouTube video player" />

* For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console or [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

  <Note>
    When generating an Elastic Cloud service instance's API key through the Elasticsearch Service web console, be sure after you click **Create API key** to select **Encoded** to get the API key's value.
    (The **Encoded** API key value is the Base64 encoding of the UTF-8 representation of the API key's ID and key value joined by a colon \[:].)
  </Note>

* For self-managed Elasticsearch, you will need:

  * The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).

  * If you're using basic authentication to the self-managed instance, the user's name and password.

  * If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

    <Note>
      When generating an API key for a self-managed Elasticsearch instance by calling [POST /\_security/api\_key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key), be sure to use
      the `encoded` value in the response payload. (This `encoded` value is the Base64 encoding of the UTF-8 representation of the payload's `id` and `api_key` values joined by a colon \[:]). Do not use the `api_key` value by itself.
    </Note>

  * If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

## Examples

To create an Elasticsearch source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="elasticsearch",
                  config={
                      "hosts": ["<host-url>"],
                      "es_api_key": "<es-api-key>",
                      "index_name": "<index-name>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "elasticsearch",
      "config": {
          "hosts": ["<host-url>"],
          "es_api_key": "<es-api-key>",
          "index_name": "<index-name>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  An array of Elastic Cloud instance URLs, for example `["https://<random-number>.<region>.<cloud-provider>.cloud.es.io"]`.
</ParamField>

<ParamField type="string">
  The Elastic Cloud API key for the target cluster.
</ParamField>

<ParamField type="string">
  The name of the target index in the cluster.
</ParamField>


# IBM FileNet
Source: https://docs.unstructured.io/api-reference/workflow/sources/filenet

Connect Unstructured to IBM FileNet as a source to ingest documents and content from your FileNet content repository into Unstructured.

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* An [IBM Cloud Pak for Business Automation as a Service](https://www.ibm.com/products/cloud-pak-for-business-automation) account with access to the IBM FileNet [Content Platform Engine server](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=architecture-content-platform-engine) object store to which you want to connect. IBM FileNet is a component of IBM Cloud Pak for Business Automation as a Service.

To access the information you'll need to configure the connector:

1. Log into your [account](https://www.automationcloud.ibm.com/auth/index.jsp).

2. Choose the **Navigator** tile.

   The [IBM Navigator](https://www.ibm.com/docs/en/content-navigator/3.2.0) launches in a separate browser window, and provides a view of your object stores and content. You can use the IBM Navigator views to find the information necessary to create a connection to Unstructured.

   For the URL of your IBM FileNet server:

   * The server URL displays in the browser address bar. You only need the base URL that specifies the company and domain. For example, `https://<company-name>.automationcloud.ibm.com`.

   For [object store](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=infrastructure-defining-object-stores) names and folder paths:

   * Select the folder in the left pane. The full folder path is displayed at the top of the main detail pane, in the following format: `<object-store>/<folder>/etc`.

   For the document class:

   * Right-click the document and select **Properties**.

   For the account username:

   * Right-click the profile icon on the upper right in the top menu.

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document. It also lists any users or groups that are explicitly *denied* those permissions.

For more information on how IBM FileNet uses Access Control Lists (ACLs) and Access Control Entries (ACEs) to manage permissions, see [About access rights](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=authorization-about-access-rights) in the IBM FileNet Platform documentation.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

<Warning>
  Always take into account the `deny_users` and `deny_groups` fields when determining the effective permissions a user or group has. Not doing so may result in over-granting permissions.

  Unstructured does not resolve groups into users when outputting permissions metadata.
</Warning>

Unstructured derives permissions metadata from the document's ACL, returned inline with the document's metadata via the IBM FileNet GraphQL API.

The document ACL returned by IBM FileNet includes the full effective ACL, including inherited permissions. Because of this, Unstructured does not further query for inherited permissions. For more information, see [ACE source: Default, Direct, Inherited, Template](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=rights-ace-source-default-direct-inherited-template) in the IBM FileNet Platform documentation.

Unstructured does not include the following permission values in the `permissions_data` field:

* MARKING

  Marking is a security classification system layered on top of standard ACL permissions. For more information, see [Markings overview](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=markings-overview) in the IBM FileNet Platform documentation.
* PROXY

  Permissions granted to a principal to act on behalf of another principal.

Unstructured writes a maximum of 1000 permission entries (ALLOW and DENY) per file.

### Permissions evaluation

IBM FileNet supports explicit DENY ACEs and evaluates them with DENY-wins semantics. This means that if a user has both an ALLOW and a DENY ACE for the same action, whether directly or through group membership, the DENY takes
precedence. For more information, see [Allow or Deny and order of evaluation](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=permissions-allow-or-deny-and-order-of-evaluation) in the IBM FileNet Platform documentation.

### Identifier formats

The connector takes the `granteeName` value that IBM FileNet returns and writes it directly into `users`, `groups`, `deny_users`, or `deny_groups` without any modification. The format of that identifier depends entirely on how your specific IBM FileNet instance is connected to its directory service.

* IBM FileNet SaaS (IBM Cloud Identity)

  LDAP distinguished names are used. For example:

  Users:`uid=alice.smith,cn=users,O=IBM,C=US`

  Groups: `cn=Finance,cn=groups,O=IBM,C=US`

* On-premises with Active Directory

  Several possible formats are possible. For example:

  * A CN=.../DC=... distinguished name:

    `CN=Alice Smith,OU=Staff,DC=contoso,DC=com`

  * A DOMAIN\user short name:

    `CONTOSO\alice.smith`

  * A Windows SID:

    `S-1-5-21-3623811015-3361044348-30300820-1013`

  * A userPrincipalName:

    `alice.smith@contoso.com`

If you're building a
downstream system that filters on these values, you will need to verify which format your instance
uses before writing that logic. We recommend running a sample query against your instance to check.

### Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity. This example uses IBM FileNet SaaS (IBM Cloud Identity) with LDAP distinguished names.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["uid=alice,cn=users,O=IBM,C=US"],
                            "groups": ["cn=Finance,cn=groups,O=IBM,C=US"],
                            "deny_users": ["uid=contractor,cn=users,O=IBM,C=US"],
                            "deny_groups": []
                        }
                    },
                    {
                        "update": {
                            "users": ["uid=alice,cn=users,O=IBM,C=US"],
                            "groups": [],
                            "deny_users": [],
                            "deny_groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": [],
                            "deny_users": [],
                            "deny_groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

## Examples

To create an IBM FileNet source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="filenet",
                  config={
                      "server_url": "<server-url>",
                      "object_store": "<object-store>",
                      "folder_path": "<folder-path>",
                      "document_class": "<document-class>",
                      "recursive": <true|false>,
                      "username": "<username>",
                      "password": "<password>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "filenet",
      "config": {
          "server_url": "<server-url>",
          "object_store": "<object-store>",
          "folder_path": "<folder-path>",
          "document_class": "<document-class>",
          "recursive": <true|false>,
          "username": "<username>",
          "password": "<password>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The base URL of your Content Platform Engine, containing both the IBM domain and your company's subdomain. For example, `https://<company-name>.automationcloud.ibm.com`.
</ParamField>

<ParamField type="string">
  The name of the object store to connect within the Content Platform Engine.
</ParamField>

<ParamField type="string">
  Source connector only. The path of the folder within the object store to use as the source.
</ParamField>

<ParamField type="string">
  Destination connector only. The path of the folder within the object store to use as the upload destination.
</ParamField>

<ParamField type="string">
  The class of documents to include.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to include documents contained in any subfolders.
</ParamField>

<ParamField type="string">
  The username of the IBM Cloud Pak for Business Automation as a Service account to use.
</ParamField>

<ParamField type="string">
  The password for the corresponding username.
</ParamField>


# Google Cloud Storage
Source: https://docs.unstructured.io/api-reference/workflow/sources/google-cloud



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Google Cloud Storage.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

* A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
  [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:
    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```
  * For Windows:
    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

  This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

  * `Storage Object Viewer` for bucket read access.
  * `Storage Object User` for bucket write access.
  * The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

  To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
  in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

## Examples

To create a Google Cloud Storage source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="gcs",
                  config={
                      "service_account_key": "<service-account-key>",
                      "remote_url": "<remote-url>",
                      "recursive": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "gcs",
      "config": {
          "service_account_key": "<service-account-key>",
          "remote_url": "<remote-url>",
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The contents of a service account key file, expressed as a single string without line breaks, for a Google Cloud service account that has the required access permissions to the bucket.
</ParamField>

<ParamField type="string">
  The URI for the Google Cloud Storage bucket and any target folder path within the bucket. This URI takes the format `gs://<bucket-name>[/folder-name]`.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to ingest data recursively from any subfolders, starting from the path specified by `remote_url`.
</ParamField>


# Google Drive
Source: https://docs.unstructured.io/api-reference/workflow/sources/google-drive



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Google Drive.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A [Google Cloud account](https://console.cloud.google.com).

* The **Google Drive API** enabled in the account.
  [Learn how](https://cloud.google.com/apis/docs/getting-started#enabling_apis).

* Within the account, a Google Cloud service account and its related `credentials.json` key file or its contents in JSON format.
  [Create a service account](https://developers.google.com/workspace/guides/create-credentials#create_a_service_account).
  [Create credentials for a service account](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the `credentials.json` key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:

    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```

  * For Windows:

    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* A Google Drive [shared folder](https://support.google.com/drive/answer/2375091) or [shared drive](https://support.google.com/a/users/answer/7212025).

* Give the service account access to the shared folder or shared drive. To do this, share the folder or drive with the service account's email address.
  [Learn how](https://support.google.com/drive/answer/7166529).
  [Learn more](https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Can-i-give-access-to-document-of-google-drive-to-service-account/m-p/530106).

* Get the shared folder's ID or shared drive's ID. This is a part of the URL for your Google Drive shared folder or shared drive, represented in the following URL as `{folder_id}`: `https://drive.google.com/drive/folders/{folder-id}`.

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "11111111111111111111"
                            ],
                            "groups": [
                                "22222222222222222222",
                                "33333333333333333333"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "44444444444444444444",
                                "55555555555555555555"
                            ],
                            "groups": [
                                "66666666666666666666",
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "77777777777777777777"
                            ],
                            "groups": [
                                "88888888888888888888"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Google Cloud user, use the user's ID along with the
[Admin SDK API](https://developers.google.com/workspace/admin/directory/reference/rest/v1/users/get) or the
[People API](https://developers.google.com/people/api/rest/v1/people/get) for Google Cloud.

To look up information about a particular Google Cloud group, use the group's ID along with the
[Admin SDK API](https://developers.google.com/workspace/admin/directory/reference/rest/v1/groups/get) or the
[Cloud Identity API](https://cloud.google.com/identity/docs/reference/rest/v1/groups/get) for Google Cloud.

## Examples

To create a Google Drive source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="google_drive",
                  config={
                      "drive_id": "<drive-id>",
                      "service_account_key": "<service-account-key>",
                      "extensions": [
                          "<extension>",
                          "<extension>"
                      ],
                      "recursive": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "google_drive",
      "config": {
          "drive_id": "<drive-id>",
          "service_account_key": "<service-account-key>",
          "extensions": [
              "<extension>",
              "<extension>"
          ],
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The ID for the target Google Drive folder or drive.
</ParamField>

<ParamField type="string">
  The contents of the `credentials.json` key file as a single-line string.
</ParamField>

<ParamField type="string">
  Set one or more file extension values (such as `pdf` or `docx`) to process files with only those extensions. If not specified, all extensions are included.

  <Note>
    Do not include the leading dot in the file extensions. For example, use `pdf` or `docx` instead of `.pdf` or `.docx`.
  </Note>
</ParamField>

<ParamField type="boolean">
  Set to `true` to recursively process data from subfolders within the target folder or drive.
</ParamField>


# Jira
Source: https://docs.unstructured.io/api-reference/workflow/sources/jira



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Jira.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A [Jira Cloud account](https://www.atlassian.com/try/cloud/signup?bundle=jira-software\&edition=free) or
  [Jira Data Center installation](https://confluence.atlassian.com/adminjiraserver/installing-jira-data-center-938846870.html).
* The site URL for your [Jira Data Center installation](https://confluence.atlassian.com/jirakb/find-your-site-url-to-set-up-the-jira-data-center-and-server-mobile-app-954244798.html) or Jira Cloud account.
  For Jira Cloud, open Jira in your web browser and copy the address from the browser's address bar.
  If you're unsure, check the dashboard URL, or if viewing an issue, project or board, the site URL is typically everything that comes before **but not including** `/jira`, such as
  `https://<organization>.atlassian.net`.
* To process Jira projects, provide the IDs for the target projects. To get a project's ID, sign in to your Jira Cloud account or Jira Data Center installation, and then go to the following URL: `https://<organization>.atlassian.net/rest/api/latest/project/<project-key>`,
  replacing `<organization>` with yours, and replacing `<project-key>` with the target project's key. In the
  response, look for the URL `https://<organization>.atlassian.net/rest/api/3/project/<project-id>`, where `<project-id>` is the target project's ID.
* To process Jira boards, the IDs for the target boards. To get a board's ID, sign in to your Jira Cloud account or Jira Data Center installation, and then go to the following URL: `https://<organization>.atlassian.net/rest/agile/1.0/board?projectKeyOrId=<project-key-or-id>`,
  replacing `<organization>` with yours, and `<project-key-or-id>` with the associated project's key or ID. In the
  response, look for the URL `https://<organization>.atlassian.net/rest/agile/1.0/board/<board-id>`, where `<board-id>` is the board's ID.
* To process Jira issues, the IDs for the target issues. To get an issue's ID, sign in to your Jia Cloud account or Jira Data Center installation, open the issue, and then look at the URL in your browser's address bar. The issue ID is the string of characters after the final slash in the URL.
* A user in your [Jira Cloud account](https://support.atlassian.com/jira-cloud-administration/docs/manage-users-groups-permissions-and-roles-in-jira-cloud/) or
  [Jira Data Center installation](https://confluence.atlassian.com/adminjiraserver/create-edit-or-remove-a-user-938847025.html).
* The user must have the correct permissions in your
  [Jira Cloud account](https://support.atlassian.com/jira-cloud-administration/docs/manage-users-groups-permissions-and-roles-in-jira-cloud/) or
  [Jira Data Center installation](https://confluence.atlassian.com/jirakb/permissions-made-simple-for-jira-server-717062767.html) to
  access the target projects, boards, and issues.
* One of the following:

  * For Jira Cloud or Jira Data Center, the target user's name or email address, and password.
    [Change a Jira Cloud user's password](https://support.atlassian.com/user-management/docs/change-password-for-portal-only-customers/).
    [Change a Jira Data Center user's password](https://confluence.atlassian.com/adminjiraserver/create-edit-or-remove-a-user-938847025.html).
  * For Jira Cloud only, the target user's name or email address, and API token.
    [Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
  * For Jira Data Center only, the target user's personal access token (PAT).
    [Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).

## Examples

To create a Jira source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="jira",
                  config={
                      "url": "<url>",

                      # For password or API token authentication:
                      # "username": "<username>",
                      # "password": "<password>",

                      # For personal access token authentication:
                      # "token": "<token>",

                      "projects": [
                          "<project-id>",
                          "<project-id>"
                      ],
                      "boards": [
                          "<board-id>",
                          "<board-id>"
                      ],
                      "issues": [
                          "<issue-id>",
                          "<issue-id>"
                      ],
                      "status_filters": [
                          "<status>",
                          "<status>"
                      ],
                      "download_attachments": <True|False>,
                      "cloud": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "jira",
      "config": {
          "url": "<url>",

          # For password or API token authentication:
          "username": "<username>",
          "password": "<password">,

          # For personal access token authentication:
          "token": "<token>",
          
          "projects": [ 
              "<project-id>",
              "<project-id>"
          ],
          "boards": [ 
              "<board-id>",
              "<board-id>"
          ],
          "issues": [ 
              "<issue-id>",
              "<issue-id>"
          ],
          "status_filters": [
              "<status>",
              "<status>"
          ],
          "download_attachments": <true|false>,
          "cloud": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The URL of the Jira instance.
</ParamField>

<ParamField type="string">
  Required for password or API token authentication. The username of the Jira user.
</ParamField>

<ParamField type="string">
  Required for password or API token authentication. For password authentication, the password of the Jira user. For API token authentication, the API token of the Jira user.
</ParamField>

<ParamField type="string">
  Required for personal access token authentication. The personal access token of the Jira user.
</ParamField>

<ParamField type="string">
  The ID of a target project in Jira to access.
</ParamField>

<ParamField type="string">
  The ID of a target board in Jira to access.
</ParamField>

<ParamField type="string">
  The ID of a target issue in Jira to access.
</ParamField>

<ParamField type="string">
  The name of a status to filter Jira issues by.
</ParamField>

<ParamField type="boolean">
  Set to `true` to download attachments from Jira issues.
</ParamField>

<ParamField type="boolean">
  Set to `true` to specify using Jira Cloud or `false` to specify using Jira Data Center. The default is to use Jira Data Center.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Jira Integration in Unstructured: Build RAG Systems with Project Management Data](https://unstructured.io/blog/jira-integration-in-unstructured-build-rag-systems-with-project-management-data)


# Kafka
Source: https://docs.unstructured.io/api-reference/workflow/sources/kafka



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Kafka.

## Requirements

You will need:

* A Kafka cluster in [Confluent Cloud](https://www.confluent.io/confluent-cloud).
  ([Create a cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#create-ak-clusters).)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

  <iframe title="YouTube video player" />

* The [hostname and port number](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#view-a-ak-cluster) of the bootstrap Kafka cluster to connect to..

* The name of the topic to read messages from or write messages to on the cluster.
  [Create a topic](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).
  [Access available topics](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).

* For authentication, an [API key and secret](https://docs.confluent.io/cloud/current/security/authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.html#add-an-api-key).

## Examples

To create a Kafka source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="kafka-cloud",
                  config={
                      "bootstrap_servers": "<bootstrap-server>",
                      "port": <port>,
                      "group_id": "<group-id>",
                      "kafka_api_key": "<kafka-api-key>",
                      "secret": "<secret>",
                      "topic": "<topic>",
                      "num_message_to_consume": <num-message-to-consume>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "kafka-cloud",
      "config": {
          "bootstrap_server": "<bootstrap-server>",
          "port": <port>,
          "group_id": "<group-id>",
          "kafka_api_key": "<kafka-api-key>",
          "secret": "<secret>",
          "topic": "<topic>",
          "num_messages_to_consume": <num-messages-to-consume>
      },
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The hostname of the bootstrap Kafka cluster to connect to.
</ParamField>

<ParamField type="integer">
  The port number of the bootstrap Kafka cluster to connect to.
</ParamField>

<ParamField type="string">
  The ID of the consumer group. A consumer group is a way to allow a pool of consumers to divide the consumption of data over topics and partitions.
</ParamField>

<ParamField type="string">
  For authentication, the API key for access to the cluster.
</ParamField>

<ParamField type="string">
  For authentication, the secret for access to the cluster.
</ParamField>

<ParamField type="string">
  The name of the topic to read messages from or write messages to on the cluster.
</ParamField>

<ParamField type="integer">
  Destination connector only. The maximum number of messages to send in a single batch.
</ParamField>

<ParamField type="integer">
  Source connector only. The maximum number of messages that the consumer will try to consume.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Unstructured Platform Now Integrates with Apache Kafka in Confluent Cloud](https://unstructured.io/blog/unstructured-platform-now-integrates-with-apache-kafka-in-confluent-cloud)


# Local
Source: https://docs.unstructured.io/api-reference/workflow/sources/local



Ingest your files into Unstructured from local storage.

This local source connector is supported only for REST API clients such as `curl` and Postman. It is currently not supported by Unstructured Pipelines or the Unstructured Python SDK.

The local source connector works only with the [local destination connector](/api-reference/workflow/destinations/local). You cannot use the local source connector with any other destination connector.

For details, see the following:

1. [Create a workflow](/api-reference/workflow/workflows#create-a-workflow) that uses a local source and local destination.
2. [Run a workflow as a job](/api-reference/api/workflow/run-workflow) that uses a local source and local destination.
3. [Download a processed local file from the workflow's completed job run](/api-reference/api/job/download-job-output).


# MongoDB
Source: https://docs.unstructured.io/api-reference/workflow/sources/mongodb



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from MongoDB.

## Requirements

You will need:

<iframe title="YouTube video player" />

The MongoDB requirements for a MongoDB Atlas deployment include:

<Warning>
  For MongoDB Atlas, SCRAM-SHA-1 is not supported for authentication. This means that cluster types that only
  include SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, and **Serverless**, are **not** supported.
  Unstructured only supports SCRAM-SHA-256 for MongoDB Atlas, which is cryptographically stronger than SCRAM-SHA-1.

  If you try to test or use a connector that refers to a cluster type that only includes SCRAM-SHA-1, the
  operation will fail, and you will get an error message similar to the following:
  `[digital envelope routines] unsupported`.
</Warning>

* A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).

* A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). Be sure to **not**
  select a cluster type that only includes SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, or **Serverless**.

* The cluster must be reachable from your application environment, for example by adding IP addresses to your IP access list. [Learn more](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).

* The cluster must be configured to allow IP address. [Learn how](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries).

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

* The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).

* The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).

* The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).

  <Note>
    For the destination connector, Unstructured recommends that all documents in the target collection have a field
    named `record_id` with a `String` data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the collection or, in some cases, the operation could fail altogether.
  </Note>

* The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

  ```text theme={null}
  mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
  ```

  To get the connection string in MongoDB Atlas, do the following:

  1. Log in to your MongoDB Atlas console.
  2. In the sidebar, under **Databases**, click **Clusters**.
  3. Click on the cluster you want to connect to.
  4. Click **Connect**.
  5. Click **Drivers**.
  6. Under **Add your connection string into your application code**, copy the connection string.
     You can then close the **Connect** dialog in MongoDB Atlas.

     Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

  [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

## Examples

To create a MongoDB source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="mongodb",
                  config={
                      "uri": "<uri>",
                      "database": "<database>",
                      "collection": "<collection>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "mongodb",
      "config": {
          "uri": "<uri>",
          "database": "<database>",
          "collection": "<collection>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The name of the database on the target MongoDB instance.
</ParamField>

<ParamField type="string">
  The name of the collection within the database.
</ParamField>

<ParamField type="string">
  The instance connection string.
</ParamField>


# OneDrive
Source: https://docs.unstructured.io/api-reference/workflow/sources/onedrive



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from OneDrive.

## Requirements

You will need:

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* The OneDrive and SharePoint Online plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.

  The following video shows how to get a path:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials, and a username and password. Both authentication types require a Microsoft Entra ID app registration.

  <Note>
    The OneDrive connector does not support any other authentication methods, such as multifactor (MFA) or passwordless authentication.
  </Note>

  You will need to provide the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration that has access to the target OneDrive account, and
  the app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Files.ReadWrite.All` (if both reading and writing are needed)
  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
  * `Directory.Read.All`

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* Both authentication types also require the User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

* For username and password authentication, you will also need the password for the target UPN.

## Examples

To create a OneDrive source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="onedrive",
                  config={
                      "client_id": "<client-id>",
                      "user_pname": "<user-pname>",
                      "password": "<password>", # For username and password authentication.
                      "tenant": "<tenant>",
                      "authority_url": "<authority-url>",
                      "client_cred": "<client-cred>",
                      "path": "<path>",
                      "recursive": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "onedrive",
      "config": {
          "client_id": "<client-id>", 
          "user_pname": "<user-pname>",
          "password": "<password>", # For username and password authentication.
          "tenant": "<tenant>", 
          "authority_url": "<authority-url>",
          "client_cred": "<client-cred>",
          "path": "<path>",
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account.
</ParamField>

<ParamField type="string">
  The User Principal Name (UPN) for the OneDrive user account in Entra ID. This is typically the user's email address.
</ParamField>

<ParamField type="string">
  Required for username and password authentication. The password for the target UPN.
</ParamField>

<ParamField type="string">
  The directory (tenant) ID of the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The authentication token provider URL for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The client secret for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  Source connector only. The path to the target folder in the OneDrive account, starting with the account's root folder, for example `my-folder/my-subfolder`.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to recursively access files from subfolders within the specified OneDrive `path`.
</ParamField>

<ParamField type="string">
  Destination connector only. `onedrive://`, followed by the path to the target folder in the OneDrive account, starting with the account's root folder, for example `onedrive://my-folder/my-subfolder`.
</ParamField>


# OpenSearch
Source: https://docs.unstructured.io/api-reference/workflow/sources/opensearch



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from OpenSearch. This page covers configuration for both Amazon OpenSearch Service (managed domains) and Amazon OpenSearch Serverless collections.

## Requirements

You will need:

## Set up OpenSearch

Supported OpenSearch installations vary by product:

* **[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):** Non-local OpenSearch only. Local OpenSearch instances are not supported.
* **[Unstructured Ingest](/open-source/ingestion/overview):** Both local and non-local OpenSearch instances are supported.

### Set up an AWS OpenSearch Service domain

To set up an [AWS OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) domain, complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.

4. Click **Create domain**.

5. In the **Name** tile, for **Domain name**, enter some unique domain name for your new OpenSearch domain.

6. In the **Domain creation method** tile, select **Easy create**. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With **Standard create**, you must enable FGAC manually.) [Standard create method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html).

7. In the **Engine options** tile, for **Version**, AWS recommends that you select the latest version.

8. In the **Network** tile, for **Network**, select a network access method.
   For faster setup, this example uses the **Public access** method.
   [VPC access method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#prerequisites-vpc-endpoints).

9. For **IP address type**, select **Dual-stack mode**.

10. In the **Fine-grained access control** (FGAC) tile, do one of the following:

    * If you want to use an existing AWS IAM user in the AWS account as the domain's master user, then for **Master user**, select **Set IAM ARN as master user**. Then enter the IAM ARN for the master user in the **IAM ARN** box.
    * If you want to create a master user and password as the domain's master user instead, then for **Master user**, select **Create master user**. Then specify some username and password for this
      new master user by filling in the **Master username**, **Master password**, and **Confirm master password** fields. Make
      sure to save the master user's password in a secure location.

11. Click **Create**.

12. After the domain is created, you must allow Unstructured to access the domain, as follows:

    a. If the new domain's settings page is not already showing, open it as follows:
    in your Amazon OpenSearch Service console, on the sidebar, expand **Managed clusters**, and then click **Domains**. Then,
    in the list of available domains, click the name of the newly created domain.<br />
    b. On the **Security configuration** tab, click **Edit**.<br />
    c. In the **Access policy** tile, for **Domain access policy**, select **Only use fine-grained access control**.<br />
    d. Click **Clear policy**. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).<br />
    e. Click **Save changes**.

### Set up an Amazon OpenSearch Serverless collection

To set up an [Amazon OpenSearch Serverless collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-create-console.html), complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Serverless**, and then click **Dashboard**.

4. Click **Create collection**.

5. In the **Collection details** tile, for **Collection name**, enter some unique name for your new OpenSearch Serverless collection.
   Optionally, for **Description**, enter some meaningful description for your new collection.

6. For **Collection type**, select **Search**.

   <Note>
     Unstructured does not support the **Vector search** collection type. If you need vector search support, you can either continue
     with these steps to use the **Search** collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead.
     However, note that the Amazon OpenSearch Serverless **Search** collection type is not as optimal as the **Vector search** collection type.
   </Note>

7. In the **Collection creation method** tile, select **Standard create**.

8. For **Encryption**, choose an AWS KMS key type.

9. For **Network access settings**, choose an **Access type**.

10. For **Resource type**, select both **Enable access to OpenSearch endpoint** and **Enable access to OpenSearch Dashboards**.

11. Click **Next**.

12. In the **Definition method** tile, select **JSON**.

13. In the **JSON editor** box, enter the following JSON, replacing the following placeholders:

    * Replace `<collection-name>` with the name of the new OpenSearch Serverless collection.
    * Replace `<account-id>` with the target AWS account ID.
    * Replace `<user-id>` with the ID of the target AWS IAM user.

    ```json theme={null}
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    ```

14. Click **Next**.

15. For **Data access policy settings**, select **Create as a new data access policy**.

16. In the **Name and description** tile, enter some unique name and an optional description for the new data access policy.

17. Click **Next**.

18. Enter any desired index details, and click **Next** again. For example:

    a. For **Index name**, enter the name of the new index in the collection.<br />
    b. For **Automatic Semantic Enrichment fields**, click **Add**, enter `embeddings` for **Automatic Semantic Enrichment field name**, click **Add**, and click **Confirm**.<br />
    c. For **Lexical search fields**, click **Add**, enter `text` for **Field name** and select **Text** for **Data type**, click **Add**, and click **Confirm**.<br />

19. Click **Submit**.

### Set up a local OpenSearch instance

The following video shows how to set up a [local OpenSearch](https://opensearch.org/downloads.html) instance.

<iframe title="YouTube video player" />

### Get the host URL

Find your host URL in the AWS console using the steps for your OpenSearch type.

#### Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.
4. In the list of available domains, click the name of your domain.
5. In the **General information** tile, copy the value of **Domain endpoint v2 (dual stack)**.

#### Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Serverless**, and then click **Dashboard**.
4. In the list of available collections, click the name of your collection.
5. On the **Overview** tab, in the **Endpoint** tile, copy the value of **OpenSearch endpoint**.

#### Local OpenSearch instance

Your local instance URL depends on how OpenSearch is installed and configured. For guidance, see [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) in the OpenSearch documentation.

### Create a search index

The name of the search index on the instance is required.

For the destination connector, if you need to create an index and you're using a master user and password as the domain's master user, you can use for example the following `curl` command. Replace the following placeholders:

* Replace `<host>` with the instance's host URL.
* Replace `<port>` with the instance's port number, which is typically `443` (for encrypted connections, and less commonly `9200` for unencrypted connections).
* Replace `<master-username>` with the master user's name, and replace `<master-password>` with the master user's password.
* Replace `<index-name>` with the name of the new search index on the instance.
* Replace `<index-schema>` with the schema for the new search index on the instance. A schema is optional; see the explanation
  following this `curl` command for more information.

```bash theme={null}
curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
```

If you're using an existing AWS IAM user as the domain's master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the preceding `curl` command. To learn how, see [create-index](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-index.html) in the AWS CLI Command Reference.

For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

For objects in the `metadata` field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index's schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
required by Unstructured whenever you create your own index schema:

* `element_id`
* `record_id`, which is required by Unstructured for intelligent record updates.
* `type`, which is not required, but highly recommended.
* `text`
* `embeddings` if embeddings are generated; make sure to set `dimension` to the same number of dimensions as the embedding model generates.

```json theme={null}
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

See also:

* [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/)
* [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
* [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
* [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
* [Unstructured document elements and metadata](/concepts/document-elements)

### Set up master user authentication

<Note>
  If you are using Enterprise Connect on a [dedicated instance](/business/dedicated-instances/overview), you do not need **master user credentials**. Skip to the **Enterprise Connect** section on this page.
</Note>

For non-local OpenSearch instances, or if you're using basic authentication to a local OpenSearch instance, you will need the master user's name and password.

For local OpenSearch instances, if you're using certificates for authentication instead of basic authentication, you will need:

* The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
* The path to the combined private key and certificate file, or
* The paths to the separate private key and certificate files.

To learn more, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

## Examples

To create an OpenSearch source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="opensearch",
                  config={
                      "hosts": ["https://<host>:<port>"],
                      "index_name": "<index-name>",
                      "username": "<username>",
                      "password": "<password>",
                      "aws_access_key_id": "<aws-access-key-id>",
                      "aws_secret_access_key": "<aws-secret-access-key>",
                      "aws_session_token": "<aws-session-token>",
                      "fields": ["<field-name>", "<field-name>"],
                      "use_ssl": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "opensearch",
      "config": {
          "hosts": ["https://<host>:<port>"],
          "index_name": "<index-name>",
          "username": "<username>",
          "password": "<password>",
          "aws_access_key_id": "<aws-access-key-id>",
          "aws_secret_access_key": "<aws-secret-access-key>",
          "aws_session_token": "<aws-session-token>",
          "fields": ["<field-name>", "<field-name>"],
          "use_ssl": "true|false"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string[]">
  The OpenSearch instance's host URL, in the format `https://<host>:<port>`.
</ParamField>

<ParamField type="string">
  The name of the search index on the instance.
</ParamField>

<ParamField type="string">
  For basic authentication, the domain's master user's name.
</ParamField>

<ParamField type="string">
  For basic authentication, the domain's master user's password.
</ParamField>

<ParamField type="string">
  For AWS IAM user authentication, the AWS access key ID. For AWS STS authentication, a temporary AWS access key ID.
</ParamField>

<ParamField type="string">
  For AWS IAM user authentication, the AWS secret access key. For AWS STS authentication, a temporary AWS secret access key.
</ParamField>

<ParamField type="string">
  For AWS STS authentication, the temporary AWS STS session token.
</ParamField>

<Warning>
  If you are using the **IAM + Session Token** authentication method: AWS STS credentials (consisting of a temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding OpenSearch connector. You must get a new set of AWS STS credentials to replace the expired ones, which produces a new, refreshed temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token. For more information, see [Request temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html).

  After you generate refreshed temporary AWS STS credentials, you must update the OpenSearch connector's settings with the new, refreshed AWS STS credentials.
</Warning>

<ParamField type="string[]">
  Source connector only. Any specific fields to be accessed in the index.
</ParamField>

<ParamField type="boolean">
  Set to `true` if the OpenSearch instance requires an SSL connection; otherwise `false`.
</ParamField>

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an OpenSearch connector to use Enterprise Connect via the Unstructured API, first set up your AWS IAM role:

### Create the IAM role

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your OpenSearch resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

### Attach a permissions policy

To allow Unstructured to access your OpenSearch installation, you may need to attach a permissions policy to the IAM role. Whether it is required depends on your OpenSearch type and how your domain access policy is configured.

#### Amazon OpenSearch Service (managed)

A permissions policy is typically not required for a managed OpenSearch service domain that has the following:

* The domain's access policy is permissive (for example, `"Principal": "*"`)
* Fine-grained access control (FGAC) is the primary access control

Both conditions above are satisfied if you selected **Easy create** in step 6 (which enables FGAC by default) and followed step 12(d) (for the domain access policy) in the [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain) section. No identity-based permissions policy is needed, and the trust policy you attached in [Create the IAM role](#create-the-iam-role) is sufficient for Enterprise Connect to function. Skip to [Grant access to OpenSearch resources](#grant-access-to-opensearch-resources).

If either condition is not met and your domain's access policy explicitly restricts access to specific principals or actions, attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:<region>:<account-id>:domain/<domain-name>/*"
    }
  ]
}
```

| Placeholder     | Value                                                  |
| --------------- | ------------------------------------------------------ |
| `<region>`      | The AWS region of the domain, for example `us-east-1`. |
| `<account-id>`  | The 12-digit ID of your AWS account.                   |
| `<domain-name>` | The name of the OpenSearch domain.                     |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

#### Amazon OpenSearch Serverless

The OpenSearch Serverless connector requires a permissions policy. Attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "aoss:APIAccessAll",
      "Resource": "arn:aws:aoss:<region>:<account-id>:collection/<collection-id>"
    }
  ]
}
```

| Placeholder       | Value                                                      |
| ----------------- | ---------------------------------------------------------- |
| `<region>`        | The AWS region of the collection, for example `us-east-1`. |
| `<account-id>`    | The 12-digit ID of your AWS account.                       |
| `<collection-id>` | The ID of the OpenSearch Serverless collection.            |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

### Grant access to OpenSearch resources

After creating the IAM role and configuring AWS permissions, you must also authorize the role within OpenSearch itself. AWS authorization and OpenSearch authorization are separate systems.

The steps in this section configure the OpenSearch authorization for your IAM role. The process you follow depends on your OpenSearch type.

#### Amazon OpenSearch Service (managed) — Map the role inside OpenSearch

You only need to map the IAM role inside OpenSearch if fine-grained access control (FGAC) is enabled on your domain:

* If you selected **Easy create** in step 6 of [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain), FGAC was enabled automatically and these steps are required.
* If your domain has FGAC disabled, skip this section. Your IAM role setup is complete. Proceed to the next section to create your connector.

**Select or create an OpenSearch role**

An *OpenSearch role* is an internal security construct, distinct from an AWS IAM role. The IAM role ARN is mapped to an internal *OpenSearch role* by registering it as a **backend role**. This is how OpenSearch links an external AWS identity to an internal permission set. First, select or create the *OpenSearch role* that has permission to perform the actions your connector needs:

* For a quick proof-of-concept, use the built-in `all_access` *OpenSearch role*, already present in OpenSearch Dashboards under **Security** → **Roles**.
* For production, first [create a least-privilege *OpenSearch role*](https://opensearch.org/docs/latest/security/access-control/users-roles/) in OpenSearch Dashboards under **Security** → **Roles**, scoped to the specific indices your connector will access.

Once you have your *OpenSearch role*, complete the steps below to map your IAM role ARN to it. Skipping this mapping causes the connector to fail with a 403 error, even if the IAM role passes the AWS authorization checks:

```
security_exception ... no permissions for [indices:admin/aliases/get] and User [name=arn:aws:iam::...:role/...]
```

For more information about managing OpenSearch roles, see [Users and roles](https://opensearch.org/docs/latest/security/access-control/users-roles/) in the OpenSearch documentation.

**Map your IAM role ARN to the OpenSearch role**

You can add the mapping using OpenSearch Dashboards or the API:

* **OpenSearch Dashboards:** Go to **Security** → **Roles** → select the *OpenSearch role* → **Mapped users** tab → add the IAM role ARN under **Backend roles**. For more information, see [Fine-grained access control in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html) in the *AWS OpenSearch Service Developer Guide*.

* Using the **API**:

  ```bash theme={null}
  curl -XPUT "$DOMAIN/_plugins/_security/api/rolesmapping/<opensearch-role-name>" \
    -H "Content-Type: application/json" \
    -u "$ADMIN_USER:$ADMIN_PASS" \
    -d '{"backend_roles":["<assumed-role-arn>"]}'
  ```

  Replace `<opensearch-role-name>` with the name of the *OpenSearch role* that you selected or created in this section. Replace `<assumed-role-arn>` with the ARN of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role).

  <Note>
    `PUT` replaces the entire role mapping. To preserve existing backend roles, first `GET` the current mapping and include all existing entries in your `PUT` request.
  </Note>

#### Amazon OpenSearch Serverless — Add the role to a data access policy

OpenSearch Serverless does not use FGAC. Authorization is managed through data access policies attached to the collection.

<Note>
  OpenSearch Serverless limits each AWS account and region to 10 data access policies. Where possible, add the IAM role ARN to an existing policy rather than creating a new one.
</Note>

Use the AWS CLI to update the collection's data access policy:

```bash theme={null}
# 1. Find the data access policy for your collection
aws opensearchserverless list-access-policies \
  --type data \
  --region <region> \
  --resource "collection/<collection-name>"

# 2. Get the current policy and note the policyVersion value in the response
aws opensearchserverless get-access-policy \
  --type data \
  --name <policy-name> \
  --region <region>

# 3. Update the policy, adding the IAM role ARN to the Principal array
aws opensearchserverless update-access-policy \
  --type data \
  --name <policy-name> \
  --policy-version <version-from-step-2> \
  --region <region> \
  --policy file://policy.json
```

The `--policy-version` value comes from the `get-access-policy` command above (#2) and enables optimistic concurrency. The `--policy` document replaces the entire policy. Include all existing principals and rules, not just your addition.

Example `policy.json`, replacing the placeholder values:

```json theme={null}
[
  {
    "Rules": [
      {
        "Resource": ["collection/<collection-name>"],
        "Permission": ["aoss:DescribeCollectionItems"],
        "ResourceType": "collection"
      },
      {
        "Resource": ["index/<collection-name>/*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument",
          "aoss:UpdateIndex"
        ],
        "ResourceType": "index"
      }
    ],
    "Principal": ["arn:aws:iam::<account-id>:role/<role-name>"],
    "Description": "Unstructured connector access"
  }
]
```

| Placeholder         | Value                                                                                               |
| ------------------- | --------------------------------------------------------------------------------------------------- |
| `<collection-name>` | The name of the OpenSearch Serverless collection.                                                   |
| `<account-id>`      | The 12-digit ID of your AWS account.                                                                |
| `<role-name>`       | The name of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role). |

**For source-only connectors:** remove `aoss:WriteDocument`,  `aoss:UpdateIndex`, and `aoss:CreateIndex` from the `Permission` array.

<Note>
  Policy changes propagate within seconds, but may occasionally take up to a minute. If the connector returns a 403 immediately after updating the policy, wait a moment and retry before treating it as a configuration error.
</Note>

### Create the source connector with Enterprise Connect

To create a source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="opensearch",
                  config={
                      "hosts": ["https://<host>:<port>"],
                      "index_name": "<index-name>",
                      "role_arn": "<role-arn>",
                      "external_id": "<external-id>",
                      "use_ssl": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "opensearch",
      "config": {
          "hosts": ["https://<host>:<port>"],
          "index_name": "<index-name>",
          "role_arn": "<role-arn>",
          "external_id": "<external-id>",
          "use_ssl": "true|false"
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string[]">
  The OpenSearch instance's host URL, for example `https://<host>:<port>`.
</ParamField>

<ParamField type="string">
  The name of the search index on the instance.
</ParamField>

<ParamField type="string">
  The ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
</ParamField>

<ParamField type="string">
  The unique value you chose as your External ID. Must match the `sts:ExternalId` condition in the IAM role's trust policy.
</ParamField>

<ParamField type="boolean">
  Set to `true` if the OpenSearch instance requires an SSL connection; otherwise `false`.
</ParamField>


# Outlook
Source: https://docs.unstructured.io/api-reference/workflow/sources/outlook



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Outlook.

## Requirements

You will need:

<iframe title="YouTube video player" />

* The Outlook user's email address.
* A Microsoft Entra ID app registration in the same Azure account as the Outlook account. You will need
  this app registration's application (client) ID, client secret, and directory (tenant) ID. [Learn how](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app).
* The Entra ID app registration must have the following Graph API permission levels of the application (not delegated) type:

  * `Mail.Read`
  * `Mail.ReadBasic`
  * `User.Read.All`

  [Learn how](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions).

## Examples

To create an Outlook source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="outlook",
                  config={
                      "client_id": "<client-id>",
                      "authority_url": "<authority-url>",
                      "tenant": "<tenant>",
                      "client_cred": "<client-cred>",
                      "user_email": "<user-email>",
                      "outlook_folders": ["<folder-name>", "<folder-name>"],
                      "recursive": <True|False>
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "type": "outlook",
      "name": "<name>",
      "config": {
          "client_id": "<client-id>",
          "authority_url": "<authority-url>",
          "tenant": "<tenant>",
          "client_cred": "<client-cred>",
          "user_email": "<user-email>",
          "outlook_folders": ["<folder-name>","<folder-name>"],
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The application (client) ID of the Microsoft Entra ID app registration that has access to the Outlook account.
</ParamField>

<ParamField type="string">
  The authentication token provider URL for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The directory (tenant) ID of the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The client secret for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The user's email address for the target Outlook account.
</ParamField>

<ParamField type="string">
  An array of folder names to access, for example: `["Inbox","Sent"]`. Do not specify any subfolder names.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to access any and all subfolders within each of the specified top-level folders.
</ParamField>


# Overview
Source: https://docs.unstructured.io/api-reference/workflow/sources/overview



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

To use the [Unstructured API's workflow operations](/api-reference/workflow/overview) to manage source connectors, do the following:

| Task                                      | Python function         | cURL                                                                          |
| ----------------------------------------- | ----------------------- | ----------------------------------------------------------------------------- |
| Get a list of available source connectors | `sources.list_sources`  | `GET` [`/sources`](/api-reference/api/source/list-sources)                    |
| Get information about a source connector  | `sources.get_source`    | `GET` [`/sources/<connector-id>`](/api-reference/api/source/get-source)       |
| Create a source connector                 | `sources.create_source` | `POST` [`/sources`](/api-reference/api/source/create-source)                  |
| Update a source connector                 | `sources.update_source` | `PUT` [`/sources/<connector-id>`](/api-reference/api/source/update-source)    |
| Delete a source connector                 | `sources.delete_source` | `DELETE` [`/sources/<connector-id>`](/api-reference/api/source/delete-source) |

To create or update a source connector, you must also provide settings that are specific to that connector.
For the list of specific settings, see:

| Connector                                                                | Python SDK           | cURL                 |
| ------------------------------------------------------------------------ | -------------------- | -------------------- |
| [Azure](/api-reference/workflow/sources/azure-blob-storage)              | `AZURE`              | `azure`              |
| [Box](/api-reference/workflow/sources/box)                               | `BOX`                | `box`                |
| [Confluence](/api-reference/workflow/sources/confluence)                 | `CONFLUENCE`         | `confluence`         |
| [Couchbase](/api-reference/workflow/sources/couchbase)                   | `COUCHBASE`          | `couchbase`          |
| [Databricks Volumes](/api-reference/workflow/sources/databricks-volumes) | `DATABRICKS_VOLUMES` | `databricks_volumes` |
| [Dropbox](/api-reference/workflow/sources/dropbox)                       | `DROPBOX`            | `dropbox`            |
| [Elasticsearch](/api-reference/workflow/sources/elasticsearch)           | `ELASTICSEARCH`      | `elasticsearch`      |
| [Google Cloud Storage](/api-reference/workflow/sources/google-cloud)     | `GCS`                | `gcs`                |
| [Google Drive](/api-reference/workflow/sources/google-drive)             | `GOOGLE_DRIVE`       | `google_drive`       |
| [IBM FileNet](/api-reference/workflow/sources/filenet)                   | `FILNET`             | `filenet`            |
| [Jira](/api-reference/workflow/sources/jira)                             | `JIRA`               | `jira`               |
| [Kafka](/api-reference/workflow/sources/kafka)                           | `KAFKA_CLOUD`        | `kafka-cloud`        |
| [Local](/api-reference/workflow/sources/local)                           |                      |                      |
| [MongoDB](/api-reference/workflow/sources/mongodb)                       | `MONGODB`            | `mongodb`            |
| [OneDrive](/api-reference/workflow/sources/onedrive)                     | `ONEDRIVE`           | `onedrive`           |
| [OpenSearch](/api-reference/workflow/sources/opensearch)                 | `OPENSEARCH`         | `opensearch`         |
| [Outlook](/api-reference/workflow/sources/outlook)                       | `OUTLOOK`            | `outlook`            |
| [PostgreSQL](/api-reference/workflow/sources/postgresql)                 | `POSTGRES`           | `postgres`           |
| [S3](/api-reference/workflow/sources/s3)                                 | `S3`                 | `s3`                 |
| [Salesforce](/api-reference/workflow/sources/salesforce)                 | `SALESFORCE`         | `salesforce`         |
| [SharePoint](/api-reference/workflow/sources/sharepoint)                 | `SHAREPOINT`         | `sharepoint`         |
| [Slack](/api-reference/workflow/sources/slack)                           | `SLACK`              | `slack`              |
| [Snowflake](/api-reference/workflow/sources/snowflake)                   | `SNOWFLAKE`          | `snowflake`          |
| [Teradata](/api-reference/workflow/sources/teradata-sql)                 | `TERADATA`           | `teradata`           |
| [VAST](/api-reference/workflow/sources/vast)                             | `VAST`               | `vast`               |
| [Zendesk](/api-reference/workflow/sources/zendesk)                       | `ZENDESK`            | `zendesk`            |


# PostgreSQL
Source: https://docs.unstructured.io/api-reference/workflow/sources/postgresql



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from PostgreSQL.

## Requirements

You will need:

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local PostgreSQL installations are not supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local PostgreSQL installations are supported.

The following video shows how to set up [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):

<iframe title="YouTube video player" />

The following video shows how to set up [Azure Database for PostgreSQL](https://azure.microsoft.com/products/postgresql):

<iframe title="YouTube video player" />

* A PostgreSQL instance.

  * [Create an Amazon RDS for PostgreSQL instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html).
  * [Create an Azure Database for PostgreSQL server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-deploy-on-azure-free-account).
  * [Install PostgreSQL locally](https://www.postgresql.org/docs/current/tutorial-install.html).

* The host name and port number for the instance.

  * For Amazon RDS for PostgreSQL, learn how to [get the host name and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html#postgresql-endpoint).
  * For Azure Database for PostgreSQL, learn how to [get the host](https://learn.microsoft.com/azure/postgresql/flexible-server/quickstart-create-server#get-the-connection-information). The port number is `5432`.
  * For local PostgreSQL installations, these values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.
  * For other installation types, see your PostgreSQL provider's documentation.

* Depending on your network security requirements, you might need to allow access to your instance only from specific IP addresses.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

  To learn how to allow these IP address ranges, see your PostgreSQL provider's documentation, for example with
  [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) or
  [Azure Database for PostgreSQL](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-manage-firewall-portal#create-a-firewall-rule-after-server-is-created).

  <Note>
    For Amazon RDS for PostgreSQL, Amazon recommends that you set the instance's **Public access** setting to **No** by default, as this
    approach is more secure. This means that no
    resources can connect to the instance outside of the instance's associated Virtual Private Cloud (VPC) without extra configuration.
    [Learn more](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Hiding).
    [Access an Amazon RDS instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html).

    If you must enable public access, set the instance's **Public access** setting to **Yes**, and then adjust the instance's related
    security group to allow this access.
    [Learn how](https://repost.aws/en/questions/QUxemKa9u5TV6CmLiO-r5prg/lost-public-access-to-aws-rds-postgresql-instance).

    [Troubleshoot issues with connecting to Amazon RDS instances](https://repost.aws/knowledge-center/rds-connectivity-instance-subnet-vpc).
  </Note>

* A database in the instance.

  * For Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, the default database name is `postgres` unless a custom database name was specified during the instance creation process.
  * For local PostgreSQL installations, learn how to [create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* A table in the database. Learn how to [create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

  For the destination connector, the table must have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the fields `id`, `element_id`, `record_id`, `text`, (and `embeddings`, if you are using `pgvector` and generating vector embeddings), as follows.
  `type` is an optional field, but highly recommended.

  If you are using `pgvector` and generating vector embeddings, the number of dimensions (in this example, `1536`) must match the number of dimensions for the associated embedding model that you use in any related Unstructured workflows or ingestion pipelines.

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT
    );
    ```

    ```sql PostgreSQL with pgvector theme={null}
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536)
    );
    ```
  </CodeGroup>

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  For objects in the `metadata` field that Unstructured produces and that you want to store in PostgreSQL, you must create fields in your table's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```

    ```sql PostgreSQL with pgvector  theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "embeddings" is required if you are generating vector embeddings.
    --   If you are generating embeddings, the number of dimensions in "embeddings" 
    --   must match the number of dimensions for the associated embedding model 
    --   that you use in any related Unstructured workflows or ingestion pipelines.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536),
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```
  </CodeGroup>

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  See also:

  * [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
  * [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  The following video shows how to use the `psql` utility to connect to PostgreSQL, list databases, and list and create tables:

  <iframe title="YouTube video player" />

* A user in the database, and a password for the user.

  * For Amazon RDS for PostgreSQL, learn how to [create a user](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Roles.html).
  * For Azure Database for PostgreSQL, learn how to [create a user](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-create-users).
  * For local PostgreSQL installations, learn how to [create a user](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* Database access for the user.

  * For Amazon RDS for PostgreSQL, learn how to [control user access](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Access.html).
  * For Azure Database for PostgreSQL, learn how to [control user access](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For local PostgreSQL installations, learn how to [give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).
  * For other installation types, see your PostgreSQL provider's documentation.

## Examples

To create a PostgreSQL source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="postgres",
                  config={
                      "host": "<host>",
                      "database": "<database>",
                      "port": <port>,
                      "username": "<username>",
                      "password": "<password>",
                      "table_name": "<table_name>",
                      "batch_size": <batch-size>,
                      "id_column": "<id-column>",
                      "fields": [
                          "<field>",
                          "<field>"
                      ]
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "postgres",
      "config": {
          "host": "<host>",
          "database": "<database>",
          "port": "<port>",
          "username": "<username>",
          "password": "<password>",
          "table_name": "<table_name>",
          "batch_size": <batch-size>,
          "id_column": "<id-column>",
          "fields": [
              "<field>",
              "<field>"
          ]
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The host name.
</ParamField>

<ParamField type="string">
  The name of the database.
</ParamField>

<ParamField type="string">
  The port number.
</ParamField>

<ParamField type="string">
  The username.
</ParamField>

<ParamField type="string">
  The user's password.
</ParamField>

<ParamField type="string">
  The name of the table in the database.
</ParamField>

<ParamField type="integer">
  The maximum number of rows to transmit at a time.
</ParamField>

<ParamField type="string">
  Source connector only. The name of the ID column in the table.
</ParamField>

<ParamField type="string">
  Source connector only. Set one or more values, with each value representing the name of a column to process (including the specified `id_column` column). The default is all columns if not otherwise specified.
</ParamField>

## Learn more

* <Icon icon="blog" />  [PostgreSQL Integration in the Unstructured Platform](https://unstructured.io/blog/postgresql-integration-in-the-unstructured-platform)


# S3
Source: https://docs.unstructured.io/api-reference/workflow/sources/s3



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Amazon S3.

## Requirements

You will need:

The following video shows how to fulfill the minimum set of Amazon S3 requirements:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
  [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

  <img alt="Enable anonymous bucket access" />

* For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
  can be granted in one of the following ways:

  * Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
    To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
    Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

    <iframe title="YouTube video player" />

  * Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
    Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
    [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
    [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
    a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

    <Warning>
      AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
      generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
      You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
      a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

      To overwrite the expired credentials with the new set:

      * For Unstructured Pipelines, manually update the AWS Key, AWS Secret Key, and STS Token fields in Unstructured Pipelines
        for the corresponding S3 [source](/pipelines/sources/s3) or [destination](/pipelines/destinations/s3) connector.
      * For the Unstructured API, use the Unstructured API's workflow operations to call the
        [update source](/api-reference/api/source/update-source) or
        [update destination](/api-reference/api/destination/update-destination) connector operation
        for the corresponding S3 [source](/api-reference/workflow/sources/s3) or
        [destination](/api-reference/workflow/destinations/s3) connector.
      * For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
        corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
    </Warning>

* If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Work with user-defined metadata

User-defined metadata in S3 is metadata that you can choose to set at the time that you upload a file to an S3 bucket.
User-defined metadata is specified in S3 as a set of name-value pairs. Each name-value pair begins with `x-amz-meta-` and is
followed by a unique name.

For more information about how to add or replace user-defined metadata for a file in S3, see the following:

* [Working with object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)
* [User-defined metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata)
* [Editing object metadata in the Amazon S3 console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-object-metadata.html)

Unstructured outputs any user-defined metadata that it finds for a file into the `metadata.data_source.record_locator.metadata` field of
the document elements' output for the corresponding file. For example, if Unstructured processes a file with the user-defined metadata
`x-amz-meta-mymetadata` name set to the value `myvalue`, Unstructured outputs the following into the `metadata.data_source.record_locator.metadata` field of
the document elements' output for the corresponding file:

```json theme={null}
[
    {
        "type": "...",
        "element_id": "...",
        "text": "...",
        "metadata": {
            "data_source": {
                "record_locator": {
                    "metadata": {
                        "mymetadata": "myvalue"
                    }
                }
            }
        }
    }
]
```

## FIPS and ambient credentials

<Note>
  The following information applies to [Unstructured Business dedicated instance and in-VPC](/business/overview) deployments only.
</Note>

Unstructured Business accounts support the Federal Information Processing Standard (FIPS) for
Amazon S3. [Learn more about AWS support for FIPS](https://aws.amazon.com/compliance/fips/). Specifically,
when creating an S3 connector with the [Unstructured Pipelines](/pipelines/overview) or
[Unstructured API's workflow operations](/api-reference/workflow/overview), Unstructured Business dedicated instance and in-VPC deployments
support the use of `fips`-based
[S3 endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html) and, for authenticated access to
S3 FIPS buckets, S3 ambient credentials.

To use Unstructured Pipelines to set up an S3 source or destination connector
to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.
2. When creating the connector, for the S3 connector's **Bucket URI** field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).
3. For the **Authentication Method** field, select **Ambient Credentials**.
4. Check the box labeled **Use Ambient Credentials**.
5. Save and test the connector.

To use the Unstructured API's workflow operations to set up an S3 source or
destination connector to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.

2. When creating the connector,
   for the `config` parameter's `remote_url` field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).

3. For the `config` parameter, add an `ambient_credentials` field, and set its value to true. For example:

   <CodeGroup>
     ```python Python SDK theme={null}
     # ...
     config={
         "...": "...",
         "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
         "ambient_credentials": True
     }
     # ...
     ```

     ```bash curl theme={null}
     # ...
     {
         "...": "...", 
         "config": {
             "...": "...",
             "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
             "ambient_credentials": "true"
         }
     }
     # ...
     ```
   </CodeGroup>

4. Run your code to create the connector.

## Examples

To create a S3 source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="s3",
                  config={
                      # For anonymous authentication:
                      # "anonymous": True,

                      # For AWS access key ID with AWS secret access key authentication:
                      # "key": "<key>",
                      # "secret": "<secret>",

                      # For AWS STS token authentication:
                      # "token": "<token>",
                      # "key": "<key>",
                      # "secret": "<secret>",

                      "remote_url": "<remote_url>",
                      "endpoint_url": "<endpoint-url>",
                      "recursive": <True|False>  # Boolean: True or False, no quotes
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "s3",
      "config": {
          # For anonymous authentication:
          "anonymous": true,

          # For AWS access key ID with AWS secret access key authentication:
          "key": "<key>", 
          "secret": "<secret>", 

          # For AWS STS token authentication:
          "token": "<token>", 
          "key": "<key>", 
          "secret": "<secret>", 

          "remote_url": "<remote_url>",
          "endpoint_url": "<endpoint-url>", 
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  For AWS access key ID or AWS STS token authentication, the AWS access key ID for the authenticated IAM user.
</ParamField>

<ParamField type="string">
  For AWS access key ID or AWS STS token authentication, the AWS secret access key corresponding to the AWS access key ID.
</ParamField>

<ParamField type="string">
  For AWS STS token authentication, the AWS STS session token for temporary access.
</ParamField>

<ParamField type="string">
  A custom URL, if connecting to a non-AWS S3 bucket.
</ParamField>

<ParamField type="string">
  The S3 URI to the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket’s root) or `s3://my-bucket/my-folder/`.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to access subfolders within the bucket.
</ParamField>

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an S3 connector to use Enterprise Connect via the Unstructured API, first set up your AWS IAM role:

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your S3 resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

3. Attach a permissions policy to the role that grants access to your S3 bucket:

   * **Source connector:** at minimum `s3:GetObject`, `s3:ListBucket`
   * **Destination connector:** at minimum `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`

   Scope the policy to the specific bucket or prefix your connector will access.

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

### Create the source connector with Enterprise Connect

To create a source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="s3",
                  config={
                      "remote_url": "<remote-url>",
                      "role_arn": "<role-arn>",
                      "external_id": "<external-id>",
                      "recursive": <True|False>  # Boolean: True or False, no quotes
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "s3",
      "config": {
          "remote_url": "<remote-url>",
          "role_arn": "<role-arn>",
          "external_id": "<external-id>",
          "recursive": <true|false>
      }
  }'
  ```
</CodeGroup>

Replace the preceding placeholders as follows.

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The S3 URI to the bucket or folder, for example `s3://my-bucket/` or `s3://my-bucket/my-folder/`.
</ParamField>

<ParamField type="string">
  The ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
</ParamField>

<ParamField type="string">
  The unique value you chose as your External ID. Must match the `sts:ExternalId` condition in the IAM role's trust policy.
</ParamField>

<ParamField type="boolean">
  Source connector only. Set to `true` to access files in subfolders within the bucket.
</ParamField>


# Salesforce
Source: https://docs.unstructured.io/api-reference/workflow/sources/salesforce



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Salesforce.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A [Salesforce account](https://developer.salesforce.com/signup).

* Your Salesforce username. To get this username, do the following:

  1. Log in to your Salesforce account.
  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.
  3. In the **Home** tab, under **Administration**, expand **Users**, and then click **Users**.
  4. Note the **User Name** value (*not* the **Name** value) for your user.

* The names of the Salesforce categories (objects) that you want to access, specified as a comma-separated list.
  Available categories include `Account`, `Campaign`, `Case`, `EmailMessage`, and `Lead`.

* A Salesforce connected app in your Salesforce account.

  If you do not already have a Salesforce connected app, to create one, start by creating or getting the certificate (`.crt`) and private key (`.pem`) files
  that you will associate with the connected app.

  If you do not have the certificate and private key files, you can use the `openssl` utility on your local machine to create your own
  private key and self-signed certificate, if your organization allows, by running the following commands, one command at a time:

  ```bash theme={null}
  openssl genrsa -out MyPrivateKey.pem -traditional
  openssl req -new -key MyPrivateKey.pem -out MyCertificateSigningRequest.csr
  openssl x509 -req -in MyCertificateSigningRequest.csr -signkey MyPrivateKey.pem -out MyCertificate.crt -days 365
  ```

  Of course, you can change these preceding example filenames as needed. Be sure to store these generated files in a secure location.

  To create a Salesforce connected app, do the following:

  1. Log in to your Salesforce account.

  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.

  3. In the **Home** tab, under **Platform Tools**, expand **Apps**, and then click **App Manager**.

  4. Click **New Connected App**.

  5. With **Create a Connected App** selected, click **Continue**.

  6. At a minimum, fill in the following, and then click **Save**:

     * **Connected App Name**

     * **API Name** (can be the same as **Connected App Name**, but do not use spaces or punctuation)

     * **Contact Email**

     * Under **API (Enable OAuth Settings)**, check **Enable OAuth Settings**.

     * For **Callback URL**, entering `https://localhost` is okay if you won't be using this connected app for other special authentication scenarios.

     * Check **Use digital signatures**, click **Choose File**, and browse to and select your certificate (`.crt`) file.

     * For **Selected OAuth Scopes**, move the following entries from the **Available OAuth Scopes** list to the **Selected OAuth Scopes** list:

       * **Manage user data via APIs (api)**
       * **Perform requests on your behalf at any time (refresh\_token, offline\_access)**

     * Uncheck **Require Proof Key for Code Exchange (PKCE) Extension for Supported Authorization Flows**.

     * Leave **Require Secret for Web Server Flow** checked.

     * Leave **Require Secret for Refresh Token Flow** checked.

     * Check **Enable Authorization Code and Credentials Flow**.

  7. On the connected app's details page, click **Manage**, click **Edit Policies**, set the following under **OAuth Policies**, and then click **Save**:

     * Set **Permitted Users** to **All users may self-authorize**.
     * Set **IP Relaxation** to **Relax IP restrictions**.
     * Set **Refresh Token Policy** to **Refresh token is valid until revoked**.

* The OAuth consumer key (client ID) for the Salesforce connected app.

  To get the Salesforce connected app's consumer key, do the following:

  1. Log in to your Salesforce account.
  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.
  3. In the **Home** tab, under **Platform Tools**, expand **Apps**, and then click **App Manager**.
  4. In the list of apps, click the arrow next to the target connected app, and click **View**.
  5. Click **Manage Consumer Details**.
  6. Complete the on-screen security verification.
  7. Note the **Consumer Key** value.

* You must use your Salesforce account to do a one-time approval of the Salesforce connected app by using its consumer key and callback URL. To do this, while you are logged in to your
  Salesforce account, browse to the following URL, replacing `<client-id>` with the consumer key value. This URL assumes that the callback URL
  is `https://localhost`:

  ```
  https://login.salesforce.com/services/oauth2/authorize?response_type=code&client_id=<client-id>&redirect_uri=https%3A%2F%2Flocalhost
  ```

* To ensure maximum compatibility across Unstructured service offerings, you should give the contents of the private key (`.pem`) file to Unstructured as
  a string that contains the contents of the file (*not* the private key file itself).\
  To print this string suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-private-key-file>` with the path to the private key file.

  * For macOS or Linux:

    ```bash theme={null}
    cat <path-to-private-key-file>
    ```

  * For Windows:

    ```text theme={null}
    Get-Content <path-to-private-key-file>
    ```

## Examples

To create a Salesforce source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="salesforce",
                  config={
                      "username": "<user-name>",
                      "consumer_key": "<consumer-key>",
                      "private_key": "<private-key>",
                      "categories": [
                          "<category>",
                          "<category>"
                      ]
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "salesforce",
      "config": {
          "username": "<user-name>",
          "consumer_key": "<consumer-key>",
          "private_key": "<private-key>",
          "categories": [
              "<category>",
              "<category>"
          ]
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The Salesforce username that has access to the required Salesforce categories.
</ParamField>

<ParamField type="string">
  The consumer key (client ID) for the Salesforce connected app.
</ParamField>

<ParamField type="string">
  The contents of the private key (PEM) associated with the consumer key for the Salesforce connected app, expressed as a single-line string.
</ParamField>

<ParamField type="string">
  An array of category values to process. The default categories are `Account`, `Campaign`, `Case`, `EmailMessage`, and `Lead`.
</ParamField>


# SharePoint
Source: https://docs.unstructured.io/api-reference/workflow/sources/sharepoint



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from SharePoint.

## Requirements

You will need:

<Note>
  If you are setting up the SharePoint connector for the first time, you can skip past this note.

  Previous versions of the SharePoint connector relied on SharePoint app principals for authentication. Current versions of the
  SharePoint connector no longer support these SharePoint app principals. Microsoft deprecated support for Share Point app principals on November 27, 2023.
  SharePoint app principals will no longer work for SharePoint tenants that were created on or after November 1, 2024, and they will stop working
  for all SharePoint tenants as of April 2, 2026. [Learn more](https://learn.microsoft.com/sharepoint/dev/sp-add-ins/retirement-announcement-for-azure-acs).

  Current versions of the SharePoint connector now rely on Microsoft Entra ID app registrations for authentication.

  To migrate from SharePoint app princpals to Entra ID app regisrations, replace the following settings in your existing SharePoint connector,
  as listed in the requirements following this note:

  * Replace the deprecated SharePoint app principal's application client ID value with your replacement Entra ID app registration's **Application (client) ID** value.
  * Replace the deprecated SharePoint app principal's client secret value with your replacement Entra ID app registration's **Client secret** value.
  * Add your replacement Entra ID app registration's **Directory (tenant) ID** value, token authority URL value, and the correct set of Microsoft Graph access permissions for SharePoint Online.

  If you need migration help, get assistance from our [Slack community](https://short.unstructured.io/pzw05l7), or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Note>

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  (Even if you only plan to use SharePoint Online, you still need a plan that includes OneDrive, because the SharePoint connector is built on OneDrive technology.)
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* The SharePoint Online and OneDrive plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The SharePoint Online site base URL. This URL must meet the following requirements:

  1. Starts with `https://`.
  2. Followed by any number of uppercase or lowercase letters, digits, or hyphens (`-`). This portion of the URL must contain at least
     one dot segment after the first domain label; for example, `.com`, `.sharepoint.com`, or `.co.uk`.
  3. Followed optionally by `/sites/` or `/teams/` and ends with any name that can include any number of
     uppercase or lowercase letters, digits, hyphens (`-`), or underscores (`_`); for example, `/sites/my-site` or `/teams/my_team`.

  Some examples of valid site URLs include:

  * `https://contoso.com`
  * `https://my-company.sharepoint.com`
  * `https://example.co.uk`
  * `https://contoso.sharepoint.com/sites/engineering`
  * `https://teams.example.com/teams/marketing`

  Additionally, note the following about collection-level URLs and all site URLs:

  * Site collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com/sites/<site-collection-name>`.
  * Team collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com/teams/<team-collection-name>`.
  * Root site collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com`.
  * To process all sites within a SharePoint tenant, use a site URL with a pattern similar to `https://<tenant>-admin.sharepoint.com`.

  [Learn more](https://learn.microsoft.com/microsoft-365/community/query-string-url-tricks-sharepoint-m365).

* The display name of the SharePoint Online library to use. The default is `Documents`.

* The path to the SharePoint Online library to use. By default, the root of the target library is used.
  To start from a path other than the root, enter the path that you want to use, beginning from the root. For example, to use
  the **my-folder > my-subfolder** path in the target library, you would specify `my-folder/my-subfolder`.

  The following video shows how to get the site URL and a path within the site:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials and a username and password. Both authentication types require a
  Microsoft Entra ID app registration. You will need to provide
  the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration, and the
  app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
    [Learn more](https://learn.microsoft.com/answers/questions/2116616/service-principal-access-to-sharepoint-online).

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* The token authority URL for your Microsoft Entra ID app registration. This is typically `https://login.microsoftonline.com`

* For username and password authentication, you must also provide the User Principal Name (UPN) and its password for the OneDrive account in the Microsoft Entra ID tenant. This UPN is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
                            ],
                            "groups": [
                                "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
                                "cccccccc-cccc-cccc-cccc-cccccccccccc"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "dddddddd-dddd-dddd-dddd-dddddddddddd",
                                "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee"
                            ],
                            "groups": [
                                "ffffffff-ffff-ffff-ffff-ffffffffffff"
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "11111111-1111-1111-1111-111111111111"
                            ],
                            "groups": [
                                "22222222-2222-2222-2222-222222222222"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Microsoft Entra ID user, use the user's ID along with the
[GET /users](https://learn.microsoft.com/graph/api/user-get) operation in the Microsoft Graph API.

To look up information about a particular Microsoft Entra ID group, use the group's ID along with the
[GET /groups](https://learn.microsoft.com/graph/api/group-get) operation in the Microsoft Graph API.

## Examples

To create a SharePoint source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="sharepoint",
                  config={
                      "site": "<site>",
                      "library": "<library>",
                      "path": "<path>",
                      "recursive": <True|False>,
                      "client_id": "<client_id>",
                      "tenant": "<tenant>",
                      "authority_url": "<authority_url>",
                      "client_cred": "<client_cred>",
                      "user_pname": "<user_pname>", # For username and password authentication.
                      "password": "<password>" # For username and password authentication.
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "sharepoint",
      "config": {
          "site": "<site>",
          "library": "<library>",
          "path": "<path>",
          "recursive": <true|false>,
          "client_id": "<client-id>",
          "tenant": "<tenant>",
          "authority_url": "<authority-url>",
          "client_cred": "<client-cred>",
          "user_pname": "<user-pname>", # For username and password authentication.
          "password": "<password>" # For username and password authentication.
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The base URL of the SharePoint site to connect to.
</ParamField>

<ParamField type="string">
  The display name of the SharePoint library to use.
</ParamField>

<ParamField type="string">
  The path to use within the library. The default is the root of the target library. To use a different path, specify the correct path format as described previously in this article.
</ParamField>

<ParamField type="boolean">
  Set to `true` to recursively process data from subfolders within the target path.
</ParamField>

<ParamField type="string">
  The client ID provided by SharePoint for the app registration.
</ParamField>

<ParamField type="string">
  The **Directory (tenant) ID** for the Microsoft Entra ID app registration with the correct set of Microsoft Graph access permissions.
</ParamField>

<ParamField type="string">
  The authentication token provider URL for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  The **Client secret** for the Entra ID app registration.
</ParamField>

<ParamField type="string">
  Required for username and password authentication. The UPN for the OneDrive account in the Entra ID tenant.
</ParamField>

<ParamField type="string">
  Required for username and password authentication. The password for the target UPN.
</ParamField>


# Slack
Source: https://docs.unstructured.io/api-reference/workflow/sources/slack



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Slack.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A Slack app. Create a Slack app by following [Step 1: Creating an app](https://api.slack.com/quickstart#creating).

* The app must have the following OAuth scopes:

  * To read messages from public channels, add the scopes `channels:join` and `channels:history`.
  * To read messages from private channels, add the scope `groups:history`.

  Add these scopes to the app by following [Step 2: Requesting scopes](https://api.slack.com/quickstart#scopes).

* The app must be installed and authorized for the target Slack workspace. Install and authorize the app by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).

* The app's access token. Get this token by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).

* Add the app to the target channels in the Slack workspace. To do this from the channel, open the channel's details page, click the **Integrations** tab, click **Add apps**, and follow the on-screen directions to install the app.

* The channel ID for each target channel. To get this ID, open the channel's details page, and look for the **Channel ID** field on the **About** tab.

* The starting and ending date and time range for the channels to be processed. Supported formats include:

  * `YYYY-MM-DD`
  * `YYYY-MM-DDTHH:MM:SS`
  * `YYYY-MM-DDTHH:MM:SSZ`
  * `YYYY-MM-DD+HH:MM:SS`
  * `YYYY-MM-DD-HH:MM:SS`

<Info>
  Slack has established API rate limits that could restrict the number and frequency of messages that can be read. To
  explore possible workarounds for these cases, see your Slack representative.
</Info>

## Examples

To create a Slack source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="slack",
                  config={
                      "channels": [
                          "<channel>",
                          "<channel>"
                      ],
                      "start_date": "<start-date>",
                      "end_date": "<end-date>",
                      "token": "<token>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "slack",
      "config": {
          "channels": [
              "<channel>",
              "<channel>"
          ],
          "start-date": "<start-date>",
          "end-date": "<end-date>",
          "token": "<token>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  An array of IDs of the target Slack channels from which to read messages.
</ParamField>

<ParamField type="string">
  The date and time, in ISO 8601 format (such as `YYYY-MM-DDTHH:MM:SSZ`), from which to begin reading messages. Messages created before this timestamp are ignored. By default, all messages are read unless an end date is specified.
</ParamField>

<ParamField type="string">
  The date and time, in ISO 8601 format (such as `YYYY-MM-DDTHH:MM:SSZ`), at which to stop reading messages. Messages created after this timestamp are ignored. By default, all messages are read unless a start date is specified.
</ParamField>

<ParamField type="string">
  The access token for the Slack app that has permission to read messages from the target Slack channels.
</ParamField>


# Snowflake
Source: https://docs.unstructured.io/api-reference/workflow/sources/snowflake



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Snowflake.

## Requirements

You will need:

* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.

  <iframe title="YouTube video player" />

  To get the identifier for the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
  3. On the **Account** tab, note the value of the **Account Identifier** field.

  Alternatively, the following Snowflake query returns the current account's identifier:

  ```text theme={null}
  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  ```

* A Snowflake user, which can be a service user (recommended) or a human user.

  To create a service user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Projects > Worksheets**.

  3. Click the **+** button to create a SQL worksheet.

  4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:

     * Replace `<service-user-name>` with some name for the service user.
     * Replace `<default-role-name>` with the name of any default role for the service user to use.

     ```sql theme={null}
     CREATE USER <service-user-name>
     DEFAULT_ROLE = "<default-role-name>"
     TYPE = SERVICE
     ```

  5. Click the arrow icon to run the worksheet, which creates the service user.

  6. To get their login name, on the navigation menu, click **Admin > Users & Roles**.

  7. On the **Users** tab, in the list of available users, click the name of the target user.

  8. In the **About** tile, note the **Login Name** for the user.

  To create a human user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & roles**.
  3. Click the **Users** tab.
  4. Click **+ User**.
  5. Follow the on-screen guidance to specify the user's settings.
  6. Click **Create User**.
  7. To get their login name, on the navigation menu, click **Admin > Users & Roles**.
  8. On the **Users** tab, in the list of available users, click the name of the target user.
  9. In the **About** tile, note the **Login Name** for the user.

* A programmatic access token (PAT) for the Snowflake user.

  <iframe title="YouTube video player" />

  To create a programmatic access token (PAT) for a user:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.

  3. On the **Users** tab, in the list of available users, click the name of the target user.

  4. In the **Programmatic access tokens** tile, click the **Generate new token** button.

  5. Follow the on-screen guidance to specify the PAT's settings.

     <Warning>
       You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later.
       Once this PAT expires, the connector will stop working.
       To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and
       update your connector's settings with your new PAT's value.

       Unstructured does not notify you when a PAT is about to expire or has already expired.
       You are responsible for tracking your PATs' expiration dates and taking corrective action before they expire.
     </Warning>

  6. Click **Generate**.

  7. Copy the generated PAT's value to a secure location, as you will not be able to access it again. If you lose this PAT's value,
     you will need to repeat this procedure to generate a new, replacement one.

  The PAT will not work unless the Snowflake account also has a valid
  [network rule](https://docs.snowflake.com/en/user-guide/network-rules) along with a valid
  [network policy](https://docs.snowflake.com/en/user-guide/network-policies) attached to that rule.
  The network rule must also be activated on the Snowflake account to begin taking effect.

  To create a valid network rule:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Rules**.

  3. Click **+ Network Rule**.

  4. Enter some name for the network rule.

  5. For **Type**, select **IPv4**.

  6. For **Mode**, select **Ingress**.

  7. For **Identifiers**, next to the magnifying glass icon, enter `0.0.0.0/0`, and then press **Enter**.

     <Note>
       The `0.0.0.0/0` value allows all IP addresses to access the Snowflake account.
       You can specify a more specific IP address range if you prefer. However, this more specific IP address range
       will apply to all users, including the user for which you created the PAT.
     </Note>

  8. Click **Create Network Rule**.

  To create a valid network policy, attaching the preceding network rule to this policy at the same time:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click **+ Network Policy**.
  4. Enter some name for the network policy.
  5. Make sure **Allowed** is selected.
  6. In the **Select rule** drop-down list, select the precedingnetwork rule to attach to this network policy.
  7. Click **Create Network Policy**.

  To activate the network rule in the account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click the name of the precedingnetwork policy to activate.
  4. In the policy's side panel, click the ellipsis (three dots) icon, and then click **Activate On Account**.
  5. Click **Activate policy**.

* (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake [user's login name (not username) and the user's password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
  This user must be a human user. Passwords are not supported for service users.

* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  <iframe title="YouTube video player" />

  * To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
  * To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
  * To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
  * To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
  * To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
  3. Click the **Roles** tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  ```text theme={null}
  SHOW ROLES;
  ```

  [Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).

* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

  <iframe title="YouTube video player" />

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:

  ```text theme={null}
  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  ```

* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

  <iframe title="YouTube video player" />

  To view a list of available databases in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  ```text theme={null}
  SHOW DATABASES;
  ```

* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  ```text theme={null}
  SHOW SCHEMAS;
  ```

  The following Snowflake query returns a list of available schemas for the database named `<database_name>` in the current account:

  ```text theme={null}
  SHOW SCHEMAS IN DATABASE <database_name>;
  ```

* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand **Tables**.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named `<schema_name>` in the datbase named
  `<database_name>` in the current account:

  ```text theme={null}
  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  ```

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the columns `ID`, `ELEMENT_ID`, and `RECORD_ID`. The columns `TEXT` and `TYPE` are optional, but highly recommended.
  If you are generating embeddings, then you must also include the column `EMBEDDINGS`.

  In the following `CREATE TABLE` statement, replace the following placeholders with the appropriate values:

  * `<database_name>`: The name of the target database in the Snowflake account.
  * `<schema_name>`: The name of the target schema in the database.
  * `<number-of-dimensions>`: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search,
    this value must match the number of dimensions that you plan to have Snowflake generate or search against.

  ```sql SQL theme={null}
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      ELEMENT_ID VARCHAR,
      RECORD_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>)
  );
  ```

  For objects in the `metadata` field that Unstructured produces and that you want to store in a Snowflake table, you must create columns in your table's schema that
  follow Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  ```sql theme={null}
  -- The ID, RECORD_ID, and ELEMENT_ID columns are required.
  -- TEXT and TYPE are not required but highly recommended.
  -- EMBEDDINGS is required if embeddings are being generated.
  -- All other "metadata" columns are optional.
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      RECORD_ID VARCHAR,
      ELEMENT_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
      IS_EXTRACTED VARCHAR,
      POINTS VARCHAR,
      SYSTEM VARCHAR,
      LAYOUT_WIDTH INTEGER,
      LAYOUT_HEIGHT INTEGER,
      FILETYPE VARCHAR,
      LANGUAGES ARRAY,
      PAGE_NUMBER VARCHAR,
      IMAGE_MIME_TYPE VARCHAR,
      FILENAME VARCHAR,
      URL VARCHAR,
      RECORD_LOCATOR VARCHAR
  );
  ```

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).

## Examples

To create a Snowflake source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="snowflake",
                  config={
                      "account": "<account>",
                      "role": "<role>",
                      "user": "<user>",
                      "password": "<programmatic-access-token>",
                      "host": "<host>",
                      "port": <port>,
                      "database": "<database>",
                      "schema_": "<schema>",
                      "table_name": "<table_name>",
                      "id_column": "<id-column>",
                      "fields": [
                          "<field>",
                          "<field>"
                      ],
                      "batch_size": <batch-size> 
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "snowflake",
      "config": {
          "account": "<account>",
          "user": "<user>",
          "host": "<host>",
          "port": <port>,
          "database": "<database>",
          "schema": "<schema>",
          "role": "<role>",
          "password": "<programmatic-access-token>",
          "id_column": "<id-column>",
          "table_name": "<table_name>",
          "batch_size": <batch-size>,
          "fields": [
              "<field>",
              "<field>"
          ]
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The target Snowflake account's identifier.
</ParamField>

<ParamField type="string">
  The name of the Snowflake role that the user belongs to. This role must have the appropriate access to the target Snowflake warehouse, database, schema, and table.
</ParamField>

<ParamField type="string">
  The target Snowflake user's login name (not their username).
</ParamField>

<ParamField type="string">
  The user's programmatic access token (PAT).

  <Note>
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  </Note>
</ParamField>

<ParamField type="string">
  The hostname of the target Snowflake warehouse.
</ParamField>

<ParamField type="integer">
  The warehouse's port number.
</ParamField>

<ParamField type="string">
  The name of the target Snowflake database.
</ParamField>

<ParamField type="string">
  The name of the target Snowflake schema within the database.
</ParamField>

<ParamField type="string">
  The name of the target Snowflake table within the database's schema. For the destination connector, the default is `elements` if not otherwise specified.
</ParamField>

<ParamField type="string">
  Source connector only. An array of columns to fetch from the table. By default, all columns are fetched unless otherwise specified.
</ParamField>

<ParamField type="string">
  Source connector only. Required. The name of the column that uniquely identifies each record in the table.
</ParamField>

<ParamField type="string">
  Destination connector only. The name of the column that uniquely identifies each record in the table.
</ParamField>

<ParamField type="integer">
  The maximum number of rows to fetch for each batch.
</ParamField>


# Teradata
Source: https://docs.unstructured.io/api-reference/workflow/sources/teradata-sql



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Teradata.

## Requirements

You will need:

* A Teradata Vantage system that can be accessed by its host name or IP address.

  For example, a Teradata Vantage system in Teradata ClearScape Analytics Experience includes:

  * A Teradata ClearScape Analytics Experience account.
  * An environment in the account.
  * A Teradata Vantage database in the environment.
  * The name and password for a Teradata user who has the appropriate access to the database.

  [Learn how to create these in Teradata ClearScape Analytics Experience](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/).

* The system's corresponding host name or IP address.

  For example, you can get these values from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics Experience account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Host** value.<br />

* The name of the target database in the system. To get a list of available databases in the system, you can run a Teradata SQL query such as the following:

  ```sql theme={null}
  SELECT DatabaseName 
  FROM DBC.DatabasesV 
  ORDER BY DatabaseName;
  ```

* The name of the target table in the database. To get a list of available tables in a database, you can run a Teradata SQL query such as the following, replacing `<database-name>` with the name of the target database:

  ```sql theme={null}
  SELECT TableName
  FROM DBC.TablesV
  WHERE DatabaseName = '<database-name>' AND TableKind = 'T'
  ORDER BY TableName;
  ```

  When Unstructured writes rows to a table, the table's columns must have a schema that is compatible with Unstructured.
  Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances.
  This is because these schemas will vary based on
  your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
  any custom post-processing code that you run; and other factors.

  In any case, note the following about table schemas:

  * The following columns are always required by Unstructured: `record_id` and `element_id`.
  * The following columns are optional for Unstructured, but highly recommended: `text` and `type`.
  * The rest of the columns are optional and typically will be output by Unstructured as part of the `metadata` field.
  * If Unstructured is generating vector embeddings, the `embeddings` column is also required.

    <Warning>
      The destination connector outputs Unstructured-generated [embeddings](/concepts/embedding) that are not directly compatible
      with Teradata Enterprise Vector Store. To use embeddings with Teradata Enterprise Vector Store, Unstructured
      recommends that you choose from among the following options:

      * Define a column in your target table named `embeddings` that is of type `VARCHAR(64000)`, to store the
        Unstructured-generated embeddings. After Unstructured adds its embeddings to your `embeddings` column,
        choose from among Teradata's options to convert the `embeddings` column's `VARCHAR` values to the
        Teradata [VECTOR Data Type](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/VECTOR-Data-Type)
        yourself.
      * Omit any columns named `embedding`, `message`, or `num_tokens` from your target table. Then choose from
        among Teradata's options (such as [AI\_TextEmbeddings](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/In-Database-Analytic-Functions-for-Vector-Store))
        to have Teradata generate the embeddings for you, instead of having Unstructured generate them.
    </Warning>

  Here is an example table schema that is compatible with Unstructured. It includes all of the required and recommended columns, as
  well as a few additional columns that are typically output by Unstructured as part of the `metadata` field. Be sure to replace
  `<database-name>` with the name of the target database and `<table-name>` with the name of the target table (by Unstructured convention,
  the table name is typically `elements`, but this is not a requirement).

  ```sql theme={null}
  CREATE SET TABLE "<database-name>"."<table-name>" (
    "id" VARCHAR(64) NOT NULL,
    PRIMARY KEY ("id"),
    "record_id" VARCHAR(64),
    "element_id" VARCHAR(64),
    "text" VARCHAR(32000) CHARACTER SET UNICODE,
    "type" VARCHAR(50),
    "embeddings" VARCHAR(64000), -- Add this column only if Unstructured is generating vector embeddings.
    "last_modified" VARCHAR(50),
    "languages" VARCHAR(200),
    "file_directory" VARCHAR(500),
    "filename" VARCHAR(255),
    "filetype" VARCHAR(50),
    "record_locator" VARCHAR(1000),
    "date_created" VARCHAR(50),
    "date_modified" VARCHAR(50),
    "date_processed" VARCHAR(50),
    "permissions_data" VARCHAR(1000),
    "filesize_bytes" INTEGER,
    "parent_id" VARCHAR(64)
  )
  ```

* For the source connector, the name of the primary key column in the table (for example, a column named `id`, typically defined as `"id" VARCHAR(64) NOT NULL, PRIMARY KEY ("id")`).

* For the source connector, the names of any specific columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* For the destination connector, the name of the column in the table that uniquely identifies each record for Unstructured to perform any necessary record updates. By default convention, Unstructured expects this field to be named `record_id`.

* The name of the Teradata user who has the appropriate access to the target database.

  For example, you can get this from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Username** value.<br />

  The Teradata SQL command to get a list of available users is as follows:

  ```sql theme={null}
  SELECT UserName 
  FROM DBC.UsersV 
  ORDER BY UserName;
  ```

* The password for the user, which was set up when the user was created.

  If the user has forgotten their password, the Teradata SQL command to change a user's password is as follows, replacing `<user-name>` with the name of the user and `<new-password>` with the new password:

  ```sql theme={null}
  MODIFY USER <user-name> SET PASSWORD = '<new-password>';
  ```

  To change a user's password, you must be an administrator (such as the `DBC` user or another user with `DROP USER` privileges).

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

## Examples

To create a Teradata source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="teradata",
                  config={
                      "host": "<host>",
                      "database": "<database>",
                      "table_name": "<table-name>",
                      "batch_size": <batch-size>,
                      "id_column": "<id-column>",
                      "fields": [
                          "<column-name>",
                          "<column-name>"
                      ],
                      "user": "<user>",
                      "password": "<password>"
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "teradata",
      "config": {
          "host": "<host>",
          "database": "<database>",
          "table_name": "<table-name>",
          "batch_size": <batch-size>,
          "id_column": "<id-column>",
          "fields": [
              "<column-name>",
              "<column-name>"
          ],
          "user": "<user>",
          "password": "<password>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The hostname or IP address associated with the target Teradata Vantage database.
</ParamField>

<ParamField type="string">
  The name of the target database. By default, the default database name is used if not otherwise specified. To get the name of the default database, you can run the Teradata SQL command `SELECT DATABASE;`.
</ParamField>

<ParamField type="string">
  The name of the target table in the database.
</ParamField>

<ParamField type="integer">
  The maximum number of rows per batch.
</ParamField>

<ParamField type="string">
  Source connector only. Required. The name of the primary key column that Teradata uses to uniquely identify each record in the table.
</ParamField>

<ParamField type="string">
  Destination connector only. The name of the column that Unstructured uses to uniquely identify each record in the table for record update purposes.
</ParamField>

<ParamField type="string">
  Source connector only. An array of column names to fetch from the table. By default, all columns are fetched unless otherwise specified.
</ParamField>

<ParamField type="string">
  The name of the user who has the appropriate access to the database.
</ParamField>

<ParamField type="string">
  The password for the user.
</ParamField>


# VAST
Source: https://docs.unstructured.io/api-reference/workflow/sources/vast

Ingest your files into Unstructured from [VAST](https://www.vastdata.com/).

<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* A [VAST](https://www.vastdata.com/) user account with permissions to the bucket to which you want to connect.

* A bucket that contains the objects to which you want to connect. A bucket is a storage resource that uses Amazon's Simple Storage Service (S3) as an access protocol.

  To create a bucket, see [Creating S3 Buckets](https://kb.vastdata.com/documentation/docs/creating-s3-buckets-via-vms-1) in the *VAST Cluster Administrator's Guide*.

* The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/` (for example, `vast://my-bucket/`). If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/` (for example, `vast://my-bucket/my-folder/`).

* The URL for the VAST region in which the bucket is located (for example, `https://vast-cluster.my-company.com`).

* An enabled access and secret key pair that grants the authenticated user permission to the VAST bucket.

  To grant a user access to a bucket, and attach identity policies that define their permissions on that bucket, see [Managing S3 User Access from the VAST Web UI](https://kb.vastdata.com/documentation/docs/managing-s3-user-access-from-the-vast-web-ui-1) or [Managing S3 Access from the VAST CLI](https://kb.vastdata.com/documentation/docs/managing-s3-access-from-the-vast-cli) in the *VAST Cluster Administrator's Guide*.

  <Note>
    VAST recommends using identity policies and bucket policies to control S3 bucket access.

    * *Identity policies* are attached to users and groups, and defines the actions the users or groups can perform against buckets.
    * *Bucket policies* are attached to a view, and define actions that particular users or groups can perform against the view and files or directories under the view.

    For more information, see [Overview of S3 Access Management](https://kb.vastdata.com/documentation/docs/overview-of-s3-access-management) in the *VAST Cluster Administrator's Guide*.

    Identity and bucket policies support a subset of elements listed in [Amazon's IAM JSON Policy Reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html). For information about the required JSON format, supported elements and identity and bucket policy examples, see [Identity and Bucket Policy Reference](https://kb.vastdata.com/documentation/docs/identity-and-bucket-policy-reference-1) in the *VAST Cluster Administrator's Guide*.
  </Note>

  For source connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:ListBucket`
  * `s3:GetObject`

  For destination connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:PutObject`

VAST buckets are private by default. [Anonymous access](https://kb.vastdata.com/documentation/docs/overview-of-vast-cluster-s3-implementation-1#support-for-authenticated-or-anonymous-requests) can be explicitly enabled using bucket policies, but is not recommended.

## Examples

To create a VAST source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="vast",
                  config={
                      "remote_url": "<remote-ur>",
                      "endpoint_url": "<endpoint-url>",
                      "recursive": <true|false>,
                      "max-objects": <integer>,
                      "key": "<key>",
                      "secret": "<secret>"    
                  }
              )
          )
      )

      print(response.source_connector_information)
  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "vast",
      "config": {
          "remote_url": "<remote-ur>",
          "endpoint_url": "<endpoint-url>",
          "recursive": <true|false>,
          "max-objects": <integer>,
          "key": "<key>",
          "secret": "<secret>"
      }
  }'
  ```
</CodeGroup>

## Configuration settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/`. For example: `vast://my-bucket/`)

  If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/`. For example: `vast://my-bucket/my-folder/`

  <Note>
    For destination connectors, the relative directory structure of files is preserved when files are uploaded. Also, files are overwritten if they already exist at the destination.
  </Note>
</ParamField>

<ParamField type="boolean">
  Source connectors only. Set to `true` to access sub-folders within the bucket.
</ParamField>

<ParamField type="integer">
  Source connectors only. The maximum number of objects to index.
</ParamField>

<ParamField type="string">
  The VAST storage endpoint URL. For example, `https://vast-cluster.my-company.com`.
</ParamField>

<ParamField type="string">
  The access key ID for the authenticated user.
</ParamField>

<ParamField type="string">
  The secret access key corresponding to the preceding access key ID.
</ParamField>


# Zendesk
Source: https://docs.unstructured.io/api-reference/workflow/sources/zendesk



<Note>
  First time creating a connector? [Read this first](/api-reference/workflow/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Zendesk.

## Requirements

You will need:

<iframe title="YouTube video player" />

* A [Zendesk account](https://www.zendesk.com/register/).
* Your [Zendesk subdomain](https://support.zendesk.com/hc/en-us/articles/4409381383578-Where-can-I-find-my-Zendesk-subdomain), for example, the `<organization>` part of `<organization>.zendesk.com`.
* The login email address for your Zendesk account.
* An [API token](https://support.zendesk.com/hc/en-us/articles/4408889192858-Managing-access-to-the-Zendesk-API#topic_tcb_fk1_2yb) (not an OAuth token) for your login email address.

## Examples

To create a Zendesk source connector, see the following examples.

For more information on working with source connectors using the Unstructured API, see [Source endpoints](/api-reference/api/source/source-apis).

<CodeGroup>
  ```python Python SDK theme={null}
  import os

  from unstructured_client import UnstructuredClient
  from unstructured_client.models.operations import CreateSourceRequest
  from unstructured_client.models.shared import CreateSourceConnector

  with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
      response = client.sources.create_source(
          request=CreateSourceRequest(
              create_source_connector=CreateSourceConnector(
                  name="<name>",
                  type="zendesk",
                  config={
                      "subdomain": "<subdomain>",
                      "email": "<email>",
                      "api_token": "<api-token>",
                      "item_type": "<item-type>",
                      "batch_size": <batch-size>
                  }
              )
          )
      )

      print(response.source_connector_information)

  ```

  ```bash curl theme={null}
  curl --request 'POST' --location \
  "$UNSTRUCTURED_API_URL/sources" \
  --header 'accept: application/json' \
  --header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  --header 'content-type: application/json' \
  --data \
  '{
      "name": "<name>",
      "type": "zendesk",
      "config": {
          "subdomain": "<subdomain>",
          "email": "<email>",
          "api-token": "<api-token>",
          "item-type": "<item-type>",
          "batch-size": <batch-size>
      }
  }'
  ```
</CodeGroup>

## Configuration Settings

Replace the preceding placeholders as follows:

<ParamField type="string">
  A unique name for this connector.
</ParamField>

<ParamField type="string">
  The subdomain of your Zendesk site, for example the `<organization>` part of `<organization>.zendesk.com`.
</ParamField>

<ParamField type="string">
  The email address of the user account that has access to the Zendesk subdomain.
</ParamField>

<ParamField type="string">
  The API token (not an OAuth token) for the user account. For more information, see [Managing access to the Zendesk API](https://support.zendesk.com/hc/en-us/articles/226022787-Generating-a-new-API-token).
</ParamField>

<ParamField type="string">
  The type of item to parse. Allowed values include `tickets` and `articles`.
</ParamField>

<ParamField type="integer">
  The maximum number of items to structure per batch.
</ParamField>

## Learn more

* <Icon icon="blog" />  [Unlock Customer Support Knowledge: Unstructured Integrates with Zendesk](https://unstructured.io/blog/unlock-customer-support-knowledge-unstructured-integrates-with-zendesk)


# Workflows
Source: https://docs.unstructured.io/api-reference/workflow/workflows

Build and manage Unstructured workflows with the Python SDK or REST API. Create, update, run, and delete workflows that process and transform documents from source to destination.

To use the [Unstructured API's workflow operations](/api-reference/workflow/overview) to manage workflows, do the following:

| Task                                       | Python function             | curl                                                                               |
| ------------------------------------------ | --------------------------- | ---------------------------------------------------------------------------------- |
| Get a list of available workflows          | `workflows.list_workflows`  | `GET` [`/workflows`](/api-reference/api/workflow/list-workflows)                   |
| Get information about a workflow           | `workflows.get_workflow`    | `GET` [`/workflows/<workflow-id>`](/api-reference/api/workflow/get-workflow)       |
| Create a workflow                          | `workflows.create_workflow` | `POST` [`/workflows`](/api-reference/api/workflow/create-workflow)                 |
| Run a workflow manually                    | `workflows.run_workflow`    | `POST` [`/workflows/<workflow-id>/run`](/api-reference/api/workflow/run-workflow)  |
| Update a workflow                          | `workflows.update_workflow` | `PUT` [`/workflows/<workflow-id>`](/api-reference/api/workflow/update-workflow)    |
| Delete a workflow                          | `workflows.delete_workflow` | `DELETE` [`/workflows/<workflow-id>`](/api-reference/api/workflow/delete-workflow) |
| Get a list of available workflow templates |                             | `GET` [`/templates`](/api-reference/api/template/list-templates)                   |
| Get information about a workflow template  |                             | `GET` [`/templates/<template-id>`](/api-reference/api/template/get-template)       |


# AI providers
Source: https://docs.unstructured.io/business/ai-providers



<Note>
  The following information applies only to **dedicated instance** and **in-VPC** deployments of [Unstructured Business](/business/overview).
</Note>

Dedicated instance and in-VPC deployments require you to supply your own AI provider credentials. Unstructured does not provision shared API keys — you bring access credentials from the AI provider of your choice. This lets you use your preferred provider and apply any volume discounts you have negotiated with that provider to your Unstructured workloads.

Credentials are required to use any of the following features:

* [Partitioning](/concepts/partitioning) that uses the **VLM** strategy, or the **Auto** strategy whenever it routes documents or pages to the **VLM** strategy
* [Enrichments](/concepts/enriching/overview)
* [Structured data extraction](/concepts/structured-data-extractor/data-extractor)
* [Embedding](/concepts/embedding)

You must have an account with each AI provider you configure. Your provider bills usage directly to your account — these charges are not included in your Unstructured bill. If you do not pay these provider bills, the features that depend on that provider may stop working without prior notice.

<Tip>
  The following steps do not apply to **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** accounts.
  This is because the costs for using the models that Unstructured provides are already factored into the
  pricing structures for these account types.

  For Unstructured Business **dedicated instance** and **in-VPC** deployments, completing the following steps enables you to
  better manage and reduce your overall costs by negotiating lower overall billing rates with Unstructured for its core features, while also
  leveraging any separately negotiated volume discounts with the AI providers that you want to use.
</Tip>

## Keeping model provider traffic private

If your network policy requires AI provider traffic to stay off the public internet, use a model provider accessible through your cloud provider's private networking.

**AWS** — Use [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html) to access foundation models through VPC endpoints. Unstructured configures the Bedrock VPC endpoint on its side; you grant access via IAM policy. Traffic never leaves the AWS network. For dedicated instance setup, see [Connecting to services with AWS PrivateLink](/business/aws/aws-privatelink/connect-to-services#aws-bedrock).

**Azure** — Use [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource) to access models through a private endpoint on the Microsoft backbone. Unstructured creates a private endpoint in its VNet toward your Azure OpenAI resource. For dedicated instance setup, see [Connecting to services with Azure Private Link](/business/azure/azure-privatelink/connect-to-services#azure-openai).

## Configure an AI provider

To have Unstructured use an AI model, you must first configure the model's AI provider for each target Unstructured
[workspace](/pipelines/account/workspaces) within each corresponding Unstructured [organizational account](/pipelines/account/organizations) in
your **dedicated instance** or **in-VPC** deployment, as follows.

This procedure assumes that you have already created an account with the model's AI provider, generated access credentials for your account
with that AI provider, and obtained any other miscellaneous settings that the AI provider might also require. For
more information, see the AI provider's documentation.

To configure an AI provider for a workspace, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account to which  the target workspace belongs.
3. Next to the organizational account selector, click the workspace selector, and then select the name of the target workspace.
4. In the workspace navigation bar, click the **AI Providers** tab.
5. In the list of available AI providers, next to the target AI provider's name, click **Configure**.
6. In the **Configure AI Provider** dialog, enter your access credentials for the AI provider, along with any other
   miscellaneous settings that the AI provider might also require. To get these settings, see your AI provider's documentation.
7. Click **Save Configuration**.

The configured AI provider is now ready to be used by all workflows in this workspace.

## Update an AI provider's configuration

After you [configure an AI provider](#configure-an-ai-provider), you can no longer open that AI provider's
settings screen to make any changes to it. This is by design to prevent users from
accessing sensitive access credentials strings or other sensitive settings that they might not otherwise be allowed
to access across your organization.

To update the settings for an AI provider, do the following:

1. [Delete the target AI provider's current configuration](#delete-an-ai-provider's-configuration).
2. [Configure the target AI provider again](#configure-an-ai-provider).

## Delete an AI provider's configuration

<Warning>
  After you delete an AI provider's configuration, any workflows in the selected workspace that rely on this AI provider
  might produce unexpected results or stop working altogether.
</Warning>

To delete an AI provider's configuration, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account to which\
   the workspace with the target AI provider's configuration belongs.
3. Next to the organizational account selector, click the workspace selector, and then select the name of the target workspace.
4. In the workspace navigation bar, click the **AI Providers** tab.
5. In the list of AI providers, next to the target AI provider's name, click the ellipsis (three dots) icon.
6. Click **Delete Configuration**.
7. To confirm the deletion, click **Delete Configuration**.

You can [configure the target AI provider again](#configure-an-ai-provider) at any time.

## No API support for AI provider management

You cannot use the Unstructured API to manage AI providers. You can manage AI providers only through Unstructured Pipelines.


# Connecting to customer-managed services on AWS
Source: https://docs.unstructured.io/business/aws/aws-privatelink/connect-to-customer-managed-services



This page covers how to establish private connectivity between your dedicated instance and services running in your AWS environment that do not have native AWS PrivateLink support.

This applies to:

* **Self-hosted applications** running in your VPC, such as MongoDB or Elasticsearch
* **AWS managed services without native PrivateLink support**, such as Amazon RDS (PostgreSQL)

For AWS-managed services with native PrivateLink support, see [Connecting to AWS-managed services](/business/aws/aws-privatelink/connect-to-services).

## How it works

Services without native PrivateLink support require you to front them with a Network Load Balancer (NLB) and expose them via a VPC Endpoint Service. Unstructured then creates a VPC Endpoint in the Unstructured-managed environment to connect to your service privately.

In AWS terminology, you are the *service provider* and Unstructured is the *service consumer*.

## Information exchange

The table below uses **Order** to indicate the general sequence of the information exchange — items with the same Order value can usually be provided at the same stage. The `<DI_AWS_ACCOUNT_ID>` value is a placeholder; contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/) to get the actual value for your dedicated instance.

<Note>
  Your service must be in the same AWS region as your dedicated instance — cross-region PrivateLink is not supported. The example region code in the table below is illustrative; use the region your dedicated instance is deployed in. See [Known limitations](/business/dedicated-instances/requirements#known-limitations).
</Note>

| Order | Information Required        | Description                                                     | Example                                                        | Provided by  |
| ----- | --------------------------- | --------------------------------------------------------------- | -------------------------------------------------------------- | ------------ |
| 1     | Service Type                | The service being accessed                                      | `PostgreSQL (RDS)`, `MongoDB`, `Elasticsearch`                 | Customer     |
| 1     | Service Region              | Region where the service is hosted                              | `us-east-1`                                                    | Customer     |
| 1     | Service Port                | Port the service listens on                                     | `5432` (PostgreSQL), `27017` (MongoDB), `9200` (Elasticsearch) | Customer     |
| 2     | Unstructured AWS Account ID | Account ID to add as allowed principal on your endpoint service | `<DI_AWS_ACCOUNT_ID>`                                          | Unstructured |
| 3     | VPC Endpoint Service Name   | Service name for the endpoint service fronting the NLB          | `com.amazonaws.vpce.us-east-1.vpce-svc-0abc123`                | Customer     |
| 3     | Service Endpoint            | The endpoint URL for connection                                 | Custom DNS or endpoint service DNS                             | Customer     |

## Steps

### Step 1: Provide service information

Provide Unstructured with the service type, region, and port via support ticket. Unstructured will share the Unstructured AWS Account ID you need to complete Step 2.

### Step 2: Create a Network Load Balancer and VPC Endpoint Service

Create an NLB targeting your service, then create a VPC Endpoint Service pointing to the NLB. [Learn how](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html)

### Step 3: Allow Unstructured as a principal

Allow Unstructured's AWS Account ID as a principal on your VPC Endpoint Service so Unstructured can create a VPC Endpoint connecting to it.

**Using the AWS Console:**

1. Navigate to **VPC > Endpoint Services**.
2. Select your endpoint service.
3. Go to the **Allow principals** tab and click **Allow principals**.
4. Enter `arn:aws:iam::<UNSTRUCTURED_AWS_ACCOUNT_ID>:root` and confirm.

**Using the AWS CLI:**

```bash theme={null}
aws ec2 modify-vpc-endpoint-service-permissions \
  --service-id vpce-svc-0abc123def456789 \
  --add-allowed-principals "arn:aws:iam::<UNSTRUCTURED_AWS_ACCOUNT_ID>:root"
```

Replace `<UNSTRUCTURED_AWS_ACCOUNT_ID>` with the account ID provided by Unstructured.

### Step 4: Share your VPC Endpoint Service name with Unstructured

Provide Unstructured with your VPC Endpoint Service name and service endpoint via support ticket. [Learn how to find your endpoint service name](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html#share-endpoint-service)

Unstructured will create a VPC Endpoint in the Unstructured-managed environment targeting your service.

### Step 5: Accept Unstructured's connection request

Once Unstructured creates the VPC Endpoint, an endpoint connection request will appear on your VPC Endpoint Service. Accept it as soon as possible to avoid delays. [Learn how](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests)

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Connecting to AWS-managed services
Source: https://docs.unstructured.io/business/aws/aws-privatelink/connect-to-services



This page describes how to establish private connectivity between your dedicated instance and AWS-managed services. For self-hosted applications or AWS services that require a Network Load Balancer, see [Connecting to customer-managed services on AWS](/business/aws/aws-privatelink/connect-to-customer-managed-services).

## AWS managed services

This section covers AWS-managed services that Unstructured can access using AWS-native private networking features, without requiring you to create a customer-managed endpoint service or Network Load Balancer.

The tables below use **Order** to indicate the general sequence of the information exchange — items with the same Order value can usually be provided at the same stage. Values like `<DI_AWS_ACCOUNT_ID>` and `<UNSTRUCTURED_IAM_ROLE_ARN>` are placeholders; contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/) to get the actual values for your dedicated instance.

<Note>
  Each AWS service you connect to via PrivateLink must be in the same AWS region as your dedicated instance — cross-region PrivateLink is not supported. The example region codes in the tables below are illustrative; use the region your dedicated instance is deployed in. See [Known limitations](/business/dedicated-instances/requirements#known-limitations).
</Note>

### AWS S3 (gateway endpoint)

| Order | Information Required      | Description                          | Example                       | Owner        |
| ----- | ------------------------- | ------------------------------------ | ----------------------------- | ------------ |
| 1     | S3 Bucket Name            | Buckets Unstructured needs to access | `my-documents`                | Customer     |
| 1     | S3 Bucket Region          | Region where bucket is located       | `us-east-1`                   | Customer     |
| 2     | Unstructured IAM Role ARN | IAM Role ARN that will access S3     | `<UNSTRUCTURED_IAM_ROLE_ARN>` | Unstructured |

This section also covers **Delta Tables in Amazon S3** — the S3 Gateway Endpoint configuration is the same.

**Example S3 Bucket Policy**

You must create a bucket policy that grants Unstructured’s IAM Role access to the required S3 buckets.

For read-only access:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowUnstructuredAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<UNSTRUCTURED_IAM_ROLE_ARN>"
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<BUCKET_NAME>",
        "arn:aws:s3:::<BUCKET_NAME>/*"
      ]
    }
  ]
}
```

Use this Action clause for write access (e.g., if S3 is a destination):

```json theme={null}
{
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:ListBucket"
  ]
}
```

Replace:

* `<UNSTRUCTURED_IAM_ROLE_ARN>` — Unstructured’s IAM Role ARN (provided during setup).
* `<BUCKET_NAME>` — Your S3 bucket name.

***

### AWS Bedrock

Amazon Bedrock is accessed via AWS-provided VPC endpoints. Unstructured configures VPC endpoints in our VPC to ensure all traffic to Bedrock stays off the public internet. Access to customer-specific Bedrock resources is controlled via IAM policies.

| Order | Information Required        | Description                                    | Example                                                                                       | Owner        |
| ----- | --------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------ |
| 1     | Bedrock Region              | AWS region where Bedrock resources are located | `us-east-1`                                                                                   | Customer     |
| 1     | Model IDs / ARNs            | Foundation models or custom models to access   | `anthropic.claude-sonnet-4-5`, `arn:aws:bedrock:us-east-1:123456789012:custom-model/my-model` | Customer     |
| 2     | Unstructured AWS Account ID | Account ID to allow in IAM/resource policies   | `<DI_AWS_ACCOUNT_ID>`                                                                         | Unstructured |
| 2     | Unstructured IAM Role ARN   | IAM Role ARN that will access Bedrock          | `<UNSTRUCTURED_IAM_ROLE_ARN>`                                                                 | Unstructured |

Unstructured configures the Bedrock VPC endpoint on the Unstructured platform. You must create IAM policies that grant access to Unstructured’s IAM Role.

**Example IAM Policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowBedrockModelInvocation",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<UNSTRUCTURED_IAM_ROLE_ARN>"
      },
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": [
        "arn:aws:bedrock:<REGION>::foundation-model/anthropic.claude-sonnet-4-5-*",
        "arn:aws:bedrock:<REGION>::foundation-model/anthropic.claude-opus-4-5-*",
        "arn:aws:bedrock:<REGION>:<CUSTOMER_ACC_NO>:custom-model/*"
      ]
    }
  ]
}
```

Replace:

* `<UNSTRUCTURED_IAM_ROLE_ARN>` — Unstructured’s AWS IAM Role ARN (provided during setup).
* `<CUSTOMER_ACC_NO>` — Your AWS Account ID.
* `<REGION>` — Your Bedrock region.

***

### Amazon Managed Streaming for Apache Kafka (MSK)

Amazon MSK supports native multi-VPC private connectivity via PrivateLink. This enables Unstructured to connect to the customer’s MSK cluster (as a Kafka source) entirely within the AWS private network. You must have an MSK cluster with **Multi-VPC Connectivity** enabled.

MSK Multi-VPC Connectivity requires MSK cluster type **provisioned** (not serverless). The cluster must use TLS or SASL/TLS authentication.

| Order | Information Required        | Description                                                 | Example                                                           | Owner        |
| ----- | --------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------- | ------------ |
| 1     | MSK Cluster ARN             | ARN of the MSK cluster                                      | `arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abc-123` | Customer     |
| 1     | MSK Cluster Region          | AWS region where cluster is deployed                        | `us-east-1`                                                       | Customer     |
| 1     | Kafka Port                  | Port the brokers listen on                                  | `9094` (TLS) or `9096` (SASL/TLS)                                 | Customer     |
| 1     | Topic Name(s)               | Kafka topics Unstructured needs to read                     | `documents-raw`, `documents-processed`                            | Customer     |
| 2     | Unstructured AWS Account ID | Account ID to add as allowed principal                      | `<DI_AWS_ACCOUNT_ID>`                                             | Unstructured |
| 3     | VPC Endpoint Service Name   | Service name created when Multi-VPC Connectivity is enabled | `com.amazonaws.vpce.us-east-1.vpce-svc-0abc123`                   | Customer     |
| 3     | Bootstrap Broker Endpoints  | Private broker DNS names for the cluster                    | `b-1.mycluster.abc123.kafka.us-east-1.amazonaws.com:9094`         | Customer     |

**Enabling MSK Multi-VPC Connectivity**

Use the AWS Console:

1. Navigate to **Amazon MSK** > select your cluster.
2. Choose **Actions** > **Edit cluster connectivity**.
3. Enable **Multi-VPC connectivity**.
4. Confirm — MSK will create a VPC Endpoint Service automatically.

Use the AWS CLI:

```bash theme={null}
aws kafka update-connectivity \
  --cluster-arn "<MSK_CLUSTER_ARN>" \
  --connectivity-info '{
    "VpcConnectivity": {
      "ClientAuthentication": {
        "Tls": { "Enabled": true }
      }
    }
  }'
```

After enabling, retrieve the VPC Endpoint Service name:

```bash theme={null}
aws kafka describe-cluster \
  --cluster-arn "<MSK_CLUSTER_ARN>" \
  --query 'ClusterInfo.BrokerNodeGroupInfo.ConnectivityInfo'
```

**Adding Unstructured as an Allowed Principal**

Once Multi-VPC Connectivity is enabled, use the AWS CLI to add Unstructured’s AWS Account ID as an allowed principal on the endpoint service:

```bash theme={null}
aws ec2 modify-vpc-endpoint-service-permissions \
  --service-id <MSK_ENDPOINT_SERVICE_ID> \
  --add-allowed-principals "arn:aws:iam::<UNSTRUCTURED_AWS_ACCOUNT_ID>:root"
```

Replace:

* `<MSK_ENDPOINT_SERVICE_ID>` — The endpoint service ID created by MSK Multi-VPC Connectivity.
* `<UNSTRUCTURED_AWS_ACCOUNT_ID>` — Unstructured’s AWS Account ID (provided during setup).

***

### Amazon OpenSearch Service

Amazon OpenSearch Service supports native Interface VPC Endpoints. Unstructured creates a VPC endpoint in our VPC targeting the customer’s OpenSearch domain.

| Order | Information Required        | Description                            | Example                                              | Owner        |
| ----- | --------------------------- | -------------------------------------- | ---------------------------------------------------- | ------------ |
| 1     | OpenSearch Domain ARN       | ARN of the OpenSearch domain           | `arn:aws:es:us-east-1:123456789012:domain/my-domain` | Customer     |
| 1     | OpenSearch Domain Region    | AWS region where domain is deployed    | `us-east-1`                                          | Customer     |
| 1     | Service Port                | Port the service listens on            | `443`                                                | Customer     |
| 2     | Unstructured AWS Account ID | Account ID to add as allowed principal | `<DI_AWS_ACCOUNT_ID>`                                | Unstructured |
| 2     | Unstructured IAM Role ARN   | IAM Role that will access OpenSearch   | `<UNSTRUCTURED_IAM_ROLE_ARN>`                        | Unstructured |
| 3     | VPC Endpoint DNS            | The endpoint DNS name for connection   | `vpc-my-domain-xyz.us-east-1.es.amazonaws.com`       | Customer     |

**Example Domain Access Policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<UNSTRUCTURED_IAM_ROLE_ARN>"
      },
      "Action": [
        "es:ESHttpGet",
        "es:ESHttpHead",
        "es:ESHttpPost",
        "es:ESHttpPut",
        "es:ESHttpDelete"
      ],
      "Resource": "arn:aws:es:<REGION>:<CUSTOMER_ACC_NO>:domain/<DOMAIN_NAME>/*"
    }
  ]
}
```

Replace:

* `<UNSTRUCTURED_IAM_ROLE_ARN>` — Unstructured’s AWS Role ARN (provided during setup).
* `<CUSTOMER_ACC_NO>` — Your AWS Account ID.
* `<REGION>` — Your OpenSearch region.
* `<DOMAIN_NAME>` — Your OpenSearch domain name.

***

### AWS OpenSearch Serverless

<Note>
  OpenSearch Serverless uses a fundamentally different access model compared to OpenSearch Service. It does not use resource-based access policies. Instead, access is controlled through **data access policies** and **network access policies** tied to VPC endpoints.
</Note>

| Order | Information Required        | Description                                                      | Example                                                 | Owner        |
| ----- | --------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------- | ------------ |
| 1     | Collection Name             | Name of the OpenSearch Serverless collection                     | `my-vector-store`                                       | Customer     |
| 1     | Collection ARN              | Full ARN of the collection                                       | `arn:aws:aoss:us-east-1:123456789012:collection/abc123` | Customer     |
| 1     | Collection Endpoint         | HTTPS endpoint of the collection                                 | `abc123.us-east-1.aoss.amazonaws.com`                   | Customer     |
| 1     | Collection Region           | AWS region where collection is deployed                          | `us-east-1`                                             | Customer     |
| 2     | Unstructured AWS Account ID | Account ID to add to network access policy                       | `<DI_AWS_ACCOUNT_ID>`                                   | Unstructured |
| 2     | Unstructured IAM Role ARN   | IAM Role ARN to grant data access                                | `<UNSTRUCTURED_IAM_ROLE_ARN>`                           | Unstructured |
| 3     | VPC Endpoint ID             | VPC endpoint ID created by Unstructured for `aoss.amazonaws.com` | `vpce-0abc123def456789`                                 | Unstructured |

**Step 1: Create a Network Access Policy**

The network access policy must allow Unstructured’s VPC endpoint to access the collection. Create or update the network policy for your collection:

```json theme={null}
[
  {
    "Rules": [
      {
        "Resource": ["collection/my-vector-store"],
        "ResourceType": "collection"
      }
    ],
    "AllowFromPublic": false,
    "SourceVPCEs": ["<UNSTRUCTURED_VPC_ENDPOINT_ID>"]
  }
]
```

Using AWS CLI:

```bash theme={null}
aws opensearchserverless update-security-policy \
  --name "my-network-policy" \
  --type network \
  --policy '[{"Rules":[{"Resource":["collection/my-vector-store"],"ResourceType":"collection"}],"AllowFromPublic":false,"SourceVPCEs":["<UNSTRUCTURED_VPC_ENDPOINT_ID>"]}]'
```

**Step 2: Create a Data Access Policy**

The data access policy grants Unstructured’s IAM Role permissions to read/write the collection’s indexes.

For a vector store destination (read/write):

```json theme={null}
[
  {
    "Rules": [
      {
        "Resource": ["index/my-vector-store/*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DeleteIndex",
          "aoss:UpdateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument"
        ],
        "ResourceType": "index"
      },
      {
        "Resource": ["collection/my-vector-store"],
        "Permission": ["aoss:DescribeCollectionItems"],
        "ResourceType": "collection"
      }
    ],
    "Principal": ["<UNSTRUCTURED_IAM_ROLE_ARN>"]
  }
]
```

Using AWS CLI:

```bash theme={null}
aws opensearchserverless create-access-policy \
  --name "unstructured-access" \
  --type data \
  --policy '[{"Rules":[{"Resource":["index/my-vector-store/*"],"Permission":["aoss:CreateIndex","aoss:DeleteIndex","aoss:UpdateIndex","aoss:DescribeIndex","aoss:ReadDocument","aoss:WriteDocument"],"ResourceType":"index"},{"Resource":["collection/my-vector-store"],"Permission":["aoss:DescribeCollectionItems"],"ResourceType":"collection"}],"Principal":["<UNSTRUCTURED_IAM_ROLE_ARN>"]}]'
```

Replace:

* `<UNSTRUCTURED_VPC_ENDPOINT_ID>` — VPC Endpoint ID provided by Unstructured (from Step 2 of the information exchange).
* `<UNSTRUCTURED_IAM_ROLE_ARN>` — Unstructured’s IAM Role ARN (provided during setup).
* `my-vector-store` — Your OpenSearch Serverless collection name.

***

### AWS Databricks

Databricks on AWS supports native PrivateLink connectivity. The customer must have a Databricks Enterprise plan with a customer-managed VPC and PrivateLink enabled on their workspace.

| Order | Information Required         | Description                                                | Example                                        | Owner        |
| ----- | ---------------------------- | ---------------------------------------------------------- | ---------------------------------------------- | ------------ |
| 1     | Databricks Workspace URL     | The workspace URL                                          | `myworkspace.cloud.databricks.com`             | Customer     |
| 1     | Databricks Workspace Region  | AWS region where workspace is deployed                     | `us-east-1`                                    | Customer     |
| 1     | Private Access Level         | Whether access is at ACCOUNT or ENDPOINT level             | `ACCOUNT`, `ENDPOINT`                          | Customer     |
| 2     | Unstructured VPC Endpoint ID | VPC Endpoint ID to add to allowed list (if ENDPOINT level) | `vpce-0abc123def456789`                        | Unstructured |
| 3     | Workspace VPC Endpoint DNS   | The private endpoint DNS for the workspace                 | `myworkspace.privatelink.cloud.databricks.com` | Customer     |

**Example: Databricks Private Access Settings (ENDPOINT level)**

If using ENDPOINT level access, add Unstructured’s VPC Endpoint ID to the allowed list via the Databricks Account Console or API:

```json theme={null}
{
  "private_access_settings_name": "unstructured-access",
  "region": "<REGION>",
  "public_access_enabled": false,
  "private_access_level": "ENDPOINT",
  "allowed_vpc_endpoint_ids": [
    "<UNSTRUCTURED_VPCE_ID>"
  ]
}
```

Replace:

* `<UNSTRUCTURED_VPCE_ID>` — VPC Endpoint ID provided by Unstructured.
* `<REGION>` — Your Databricks region.

For ACCOUNT level access, no explicit endpoint allowlisting is required — any VPC endpoint registered in the Databricks account can connect.

***

## AWS Managed Services with native PrivateLink

Some AWS managed services support native PrivateLink endpoints. You must first create a VPC endpoint for the service.  Unstructured then connects to it. This pattern applies to services like **Amazon ElastiCache (Redis)** and **AWS Elasticsearch Service (legacy)**.

<Note>
  The **Order** column indicates the general sequence for the information exchange. Items with the same order value can usually be provided at the same stage.
</Note>

| Order | Information Required        | Description                            | Example                                                       | Owner        |
| ----- | --------------------------- | -------------------------------------- | ------------------------------------------------------------- | ------------ |
| 1     | Service Type                | The AWS Service being accessed         | `ElastiCache`, `Elasticsearch`                                | Customer     |
| 1     | Service Region              | Region where the service is hosted     | `us-east-1`                                                   | Customer     |
| 1     | Service Port                | Port the service listens on            | `6379` (Redis), `443` (Elasticsearch)                         | Customer     |
| 1     | Resource ARN                | ARN of the resource                    | `arn:aws:elasticache:us-east-1:123456789012:cluster/my-cache` | Customer     |
| 2     | Unstructured AWS Account ID | Account ID to add as allowed principal | `<DI_AWS_ACCOUNT_ID>`                                         | Unstructured |
| 2     | Unstructured IAM Role ARN   | IAM Role that will access the service  | `<UNSTRUCTURED_IAM_ROLE_ARN>`                                 | Unstructured |
| 3     | VPC Endpoint ID             | The service-managed VPC endpoint ID    | `vpce-0abc123def456789`                                       | Customer     |
| 3     | VPC Endpoint DNS            | The endpoint DNS name for connection   | `vpce-0abc123.us-east-1.es.amazonaws.com`                     | Customer     |


# Connector support for AWS PrivateLink
Source: https://docs.unstructured.io/business/aws/aws-privatelink/connector-support



Not all Unstructured platform connectors support AWS PrivateLink. The following table summarizes support and configuration for each connector in a dedicated instance deployment.

| Connector                                                                                                   | Private connectivity supported | Approach                                                             | Configuration details                                                                                   |
| ----------------------------------------------------------------------------------------------------------- | :----------------------------: | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Customer → Unstructured (Pipelines/API)**                                                                 |                ✅               | Interface VPC endpoint                                               | [Link](/business/aws/aws-privatelink/onboarding-di-pl#step-3-connect-to-the-unstructured-platform)      |
| **AWS S3**                                                                                                  |                ✅               | Gateway endpoint                                                     | [Link](/business/aws/aws-privatelink/connect-to-services#aws-s3-gateway-endpoint)                       |
| **AWS Bedrock** (VLM)                                                                                       |                ✅               | Interface VPC endpoint on the Unstructured side                      | [Link](/business/aws/aws-privatelink/connect-to-services#aws-bedrock)                                   |
| **AWS OpenSearch Service**                                                                                  |                ✅               | Interface VPC endpoint                                               | [Link](/business/aws/aws-privatelink/connect-to-services#aws-opensearch-service)                        |
| **AWS OpenSearch Serverless**                                                                               |                ✅               | VPC Endpoint + Data Access Policy                                    | [Link](/business/aws/aws-privatelink/connect-to-services#aws-opensearch-serverless)                     |
| **AWS MSK (Kafka)**                                                                                         |                ✅               | Multi-VPC Connectivity (native MSK support)                          | [Link](/business/aws/aws-privatelink/connect-to-services#aws-msk-kafka)                                 |
| **AWS RDS / PostgreSQL**                                                                                    |                ✅               | NLB-backed endpoint service                                          | [Link](/business/aws/aws-privatelink/connect-to-customer-managed-services)                              |
| **Databricks on AWS**                                                                                       |                ✅               | Databricks PrivateLink                                               | [Link](/business/aws/aws-privatelink/connect-to-services#aws-databricks)                                |
| **Elasticsearch (self-hosted on AWS)**                                                                      |                ✅               | NLB-backed endpoint service                                          | [Link](/business/aws/aws-privatelink/connect-to-customer-managed-services)                              |
| **MongoDB (self-hosted on AWS)**                                                                            |                ✅               | NLB-backed endpoint service                                          | [Link](/business/aws/aws-privatelink/connect-to-customer-managed-services)                              |
| **Box, Confluence, Dropbox, Google Drive, Jira, OneDrive, Outlook, Salesforce, SharePoint, Slack, Zendesk** |                ❌               | SaaS application; requires public egress                             | Requires [public egress](/business/dedicated-instances/security/security-model#network-access-controls) |
| **Google Cloud Storage**                                                                                    |                ❌               | Cross-cloud access is not supported through AWS private connectivity | Requires [public egress](/business/dedicated-instances/security/security-model#network-access-controls) |


# Configure DNS for AWS PrivateLink
Source: https://docs.unstructured.io/business/aws/aws-privatelink/dns-configuration



This page describes how to configure DNS so your dedicated instance hostname (`<companyname>.privatelink.unstructuredapp.io`) resolves to the private IPs of your AWS PrivateLink VPC Endpoint. This step happens after you create your VPC Endpoint as part of [onboarding](/business/aws/aws-privatelink/onboarding-di-pl).

Clients must connect using the dedicated instance hostname because TLS is terminated using a provider-managed certificate. The two most common setups are below — choose the one that matches your authoritative DNS.

## Choose your setup

| Your authoritative DNS is...                                           | Go to                                         |
| ---------------------------------------------------------------------- | --------------------------------------------- |
| Amazon Route 53                                                        | [You use Route 53](#you-use-route-53)         |
| An internal DNS system such as Infoblox, BIND, or Active Directory DNS | [You use your own DNS](#you-use-your-own-dns) |

<Note>
  A Zero Trust Network (such as Zscaler) is a network path, not a DNS authority. Clients connecting through a ZTN still resolve DNS through whichever DNS system you use, so the section that matches your authoritative DNS already covers them. Note that some ZTN configurations intercept or proxy DNS queries — if your ZTN performs DNS interception, verify that queries for `<companyname>.privatelink.unstructuredapp.io` are forwarded to your configured DNS system and not resolved independently by the ZTN.
</Note>

## You use Route 53

This is the simplest setup if you already use Route 53 in your AWS account.

### In-VPC clients

When you create your VPC Endpoint, enable the **Enable private DNS name** option. The Amazon-provided DNS resolver in your VPC will then return the endpoint's private IPs whenever clients query your dedicated instance hostname. No additional DNS configuration is needed. (Note: this resolution is handled by the VPC resolver — you will not see a corresponding private hosted zone in the Route 53 console.)

This requires DNS hostnames and DNS resolution to be enabled on your VPC. See [VPC DNS attributes](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating).

### On-premises clients (Direct Connect or VPN)

The private DNS resolution configured in the previous section only works from inside your VPC. If you have on-premises clients reaching the platform over Direct Connect or a VPN, bridge your on-premises DNS to Route 53 with a Resolver inbound endpoint and a conditional forwarder.

This assumes you already have a private network connection (Direct Connect or VPN) between your on-premises network and your VPC.

1. **Create a Route 53 Resolver inbound endpoint.** In the AWS console, go to **Route 53 → Resolver → Inbound endpoints**, and create an endpoint with at least two ENIs in different Availability Zones in your VPC. Note the IP address of each ENI (for example, `10.0.1.10` and `10.0.2.10`). The endpoint's security group must allow inbound TCP and UDP traffic on port 53 from your on-premises network.

2. **Add a conditional forwarder on your on-premises DNS.** Forward queries for `<companyname>.privatelink.unstructuredapp.io` to the Resolver inbound endpoint IPs. This forwarding rule covers all subdomains (for example `api.<companyname>.privatelink.unstructuredapp.io`) automatically — no additional rules are needed per subdomain. Customers with multiple dedicated instances configure one conditional forwarder per dedicated instance.

Examples for common DNS systems:

* **Windows DNS Server:** Open **DNS Manager → Conditional Forwarders → New Conditional Forwarder**. Enter `<companyname>.privatelink.unstructuredapp.io` as the DNS Domain and add both Resolver IPs.

* **BIND:** Add the following to `named.conf`, then run `rndc reload`:

  ```text theme={null}
  zone "<companyname>.privatelink.unstructuredapp.io" {
      type forward;
      forward only;
      forwarders { 10.0.1.10; 10.0.2.10; };
  };
  ```

* **Infoblox:** Go to **Data Management → DNS → Forward Zones → Add**. Enter `<companyname>.privatelink.unstructuredapp.io` as the zone name and add both Resolver IPs as forwarders.

3. **Validate** from an on-premises client. See [Validation](#validation).

## You use your own DNS

If your authoritative DNS is something other than Route 53 — for example, Infoblox, BIND, or Active Directory DNS — this is the recommended path. The same records serve clients in your VPC, on-premises clients reaching the platform over Direct Connect or VPN, and laptops routed through a Zero Trust Network. There is no separate setup per client location, and no Route 53 Resolver inbound endpoint or conditional forwarder is required.

Unstructured cannot create these records for you because your VPC Endpoint and its private IPs live in your account.

<Warning>
  **VPC instances must use your internal DNS.** This setup only works for in-VPC clients if your VPC instances are configured to query your internal DNS server rather than the default Amazon-provided resolver. Update your VPC's DHCP option set to point to your internal DNS server IPs. Without this change, EC2 instances and other in-VPC workloads will use the Amazon resolver, which has no knowledge of your internal records and will not resolve the hostname correctly.
</Warning>

### Recommended: A records to the VPC Endpoint's private IPs

Add an apex and wildcard A record on a per-dedicated-instance zone in your internal DNS. The records resolve directly to the VPC Endpoint's private IPs. The wildcard ensures all current and future subdomains — for example `api.` and `platform-api.` — resolve without additional DNS changes.

Example records for a VPC Endpoint with ENIs at `10.0.5.12` and `10.0.6.12`:

| Record                                           | Type | Target                   |
| ------------------------------------------------ | ---- | ------------------------ |
| `<companyname>.privatelink.unstructuredapp.io`   | A    | `10.0.5.12`, `10.0.6.12` |
| `*.<companyname>.privatelink.unstructuredapp.io` | A    | `10.0.5.12`, `10.0.6.12` |

A VPC Endpoint creates one ENI per Availability Zone, so most dedicated instances have two or three private IPs. List all of them on each record — DNS round-robins across them, giving you per-AZ failover.

A TTL of 300 seconds is recommended. Find the VPC Endpoint's private IPs in the AWS console: open the VPC Endpoint, go to the **Subnets** tab, and read the **IPv4 addresses** column for each ENI.

### Multiple dedicated instances

Customers with multiple dedicated instances configure a separate zone per dedicated instance. For example, a customer with development, staging, and production instances configures:

* `<companyname>-dev.privatelink.unstructuredapp.io` → A records pointing at the dev instance's VPC Endpoint private IPs
* `<companyname>-staging.privatelink.unstructuredapp.io` → A records pointing at the staging instance's VPC Endpoint private IPs
* `<companyname>-prod.privatelink.unstructuredapp.io` → A records pointing at the prod instance's VPC Endpoint private IPs

This avoids claiming the entire `privatelink.unstructuredapp.io` namespace on your internal DNS and lets you connect to additional dedicated instances later without conflicts.

## Validation

From within the target network, run the following:

```bash theme={null}
dig <companyname>.privatelink.unstructuredapp.io
dig api.<companyname>.privatelink.unstructuredapp.io
```

Both queries should return private IP addresses within your VPC CIDR range. If they return public IP addresses or fail to resolve, see [Troubleshooting](#troubleshooting).

## Troubleshooting

| Problem                                                                        | Likely cause                                                                                 | What to check                                                                                                                                                                                                                       |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dig` returns public IP addresses                                              | The client is querying public DNS instead of the DNS system you configured                   | Confirm the client's resolver settings; confirm your internal DNS is authoritative for `<companyname>.privatelink.unstructuredapp.io` (own-DNS setup) or that your conditional forwarder is in place (Route 53 + on-premises setup) |
| `dig` returns NXDOMAIN                                                         | The records are missing, or the client cannot reach your DNS                                 | Confirm the records exist in your DNS system and that the client can reach your DNS resolvers                                                                                                                                       |
| Resolution works in the VPC but fails from on-premises (Route 53 setup)        | The Route 53 Resolver inbound endpoint or on-premises conditional forwarder is misconfigured | Confirm the inbound endpoint has at least two ENIs in different AZs, that its security group allows port 53 from your on-premises network, and that the conditional forwarder points at the inbound endpoint's ENI IPs              |
| Resolution works from on-premises but fails for in-VPC clients (own-DNS setup) | VPC instances are using the Amazon resolver instead of your internal DNS                     | Update the VPC's DHCP option set to point to your internal DNS server IPs                                                                                                                                                           |
| TLS handshake fails after DNS resolves to private IPs                          | The client is connecting to a hostname not covered by the provider-managed certificate       | Confirm the client is connecting to a hostname under `<companyname>.privatelink.unstructuredapp.io`                                                                                                                                 |

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Onboarding your dedicated instance + AWS PrivateLink
Source: https://docs.unstructured.io/business/aws/aws-privatelink/onboarding-di-pl



<Note>
  The following information applies only to dedicated instance deployments with AWS PrivateLink for [Unstructured Business](/business/overview) on AWS.

  For dedicated instance deployments of Unstructured Business to AWS *without* PrivateLink, contact your Unstructured sales representative,
  or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

After your organization signs the **Business** account agreement with Unstructured, a member of the Unstructured technical enablement team will reach out to begin the deployment onboarding process.

## High-level onboarding process

| Step | Owner                   | Description                                                                             |
| :--: | ----------------------- | --------------------------------------------------------------------------------------- |
|   1  | Customer                | Provide Unstructured with the information required to provision your dedicated instance |
|   2  | Unstructured            | Build and configure your dedicated instance                                             |
|   3  | Customer                | Establish private connectivity to access Unstructured Pipelines and the API             |
|   4  | Customer + Unstructured | Connect your dedicated instance to your data sources and destinations                   |

<Note>
  Use a support ticket for all configuration data exchanges. Open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/) with the subject line **`PrivateLink Configuration Request — [Your Company Name]`**.
</Note>

## Step 1: Provide provisioning information

Provide the information listed in [Before you begin](/business/dedicated-instances/before-you-begin#what-to-prepare-before-onboarding) to Unstructured to begin provisioning your dedicated instance.

## Step 2: Unstructured provisions your dedicated instance

No action is required from you during this step. Once provisioning is complete, Unstructured will provide you with:

* The AWS Account ID your dedicated instance is hosted in
* The VPC Endpoint Service name for your dedicated instance (format: `com.amazonaws.vpce.<region>.vpce-svc-<id>`)
* The Availability Zones supported by the endpoint service

## Step 3: Connect to the Unstructured Platform

Once your dedicated instance is provisioned, establish private connectivity from your environment so your users can access Pipelines and the API.

### Information exchange

| Information                  | Description                                         | Example                                         | Provided by  |
| ---------------------------- | --------------------------------------------------- | ----------------------------------------------- | ------------ |
| Unstructured AWS Account ID  | AWS Account ID your dedicated instance is hosted in | `<DI_AWS_ACCOUNT_ID>`                           | Unstructured |
| VPC Endpoint Service Name    | Service name for the Unstructured endpoint          | `com.amazonaws.vpce.us-east-1.vpce-svc-0abc123` | Unstructured |
| Supported Availability Zones | AZs available for the endpoint service              | `us-east-1a`, `us-east-1b`                      | Unstructured |
| Customer AWS Account ID(s)   | 12-digit ID of each AWS account that needs access   | `123456789012`                                  | Customer     |

### Step 3a: Provide your AWS Account IDs

Provide Unstructured with the AWS Account IDs that need access to your dedicated instance. Unstructured will add these accounts as allowed principals on the VPC Endpoint Service.

### Step 3b: Create Interface VPC Endpoints

In each AWS account that requires access, create an Interface VPC Endpoint targeting Unstructured's VPC Endpoint Service. [Learn how](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html)

When creating the endpoint:

* Use the Endpoint Service name provided by Unstructured in Step 2
* Select subnets in the Availability Zones supported by the endpoint service
* The endpoint must be in the same AWS Region as your dedicated instance

Creating the endpoint sends a connection request to Unstructured.

### Step 3c: Unstructured approves your connection request

No action is required from you during this step. Unstructured will approve your VPC Endpoint connection request.

### Step 3d: Configure DNS

Configure DNS so your dedicated instance hostname resolves to the private IPs of your VPC Endpoint. The right approach depends on your authoritative DNS and where your clients connect from.

For setup details across the most common scenarios — clients in your AWS VPC, on-premises clients reaching the platform over Direct Connect or VPN, and clients connecting through a Zero Trust Network — see [Configure DNS for AWS PrivateLink](/business/aws/aws-privatelink/dns-configuration).

## Step 4: Connect to your data sources and destinations

Once your users can access the Unstructured Platform, connect the platform to your data sources and destinations. The approach depends on the type of service.

* **AWS-managed services** (S3, Bedrock, MSK, OpenSearch, Databricks): See [Connecting to AWS-managed services](/business/aws/aws-privatelink/connect-to-services)
* **Customer-managed services on AWS** (self-hosted applications such as MongoDB or Elasticsearch, or AWS services that require a Network Load Balancer): See [Connecting to customer-managed services on AWS](/business/aws/aws-privatelink/connect-to-customer-managed-services)

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Enterprise Connect for AWS
Source: https://docs.unstructured.io/business/aws/enterprise-connect

Enterprise Connect lets Unstructured connectors access AWS resources by assuming a customer-configured IAM role with temporary, scoped credentials.

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to have it enabled on your instance.
</Note>

## Enterprise Connect credential model

Enterprise Connect is an authentication method that provides workspace-level credential management for AWS connectors. During a workflow run, the connector uses AWS Security Token Service (STS) to assume an IAM role in your AWS account and receive temporary credentials. No credentials are stored; they expire automatically.

## Supported AWS connectors

Enterprise Connect supports the following AWS connectors:

* AWS S3
* AWS OpenSearch

## How Enterprise Connect works

Enterprise Connect requires setup in your AWS account. You create an IAM role and attach a trust policy that permits Unstructured's service role to assume it. The trust policy includes an external ID, a unique value you generate and provide. Because only you know this value, only your workspace can trigger the role assumption, preventing the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html).

When a workflow runs, the connector uses the AWS STS [`AssumeRole`](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) API to assume your IAM role and receive temporary, least-privilege credentials. Unstructured uses the credentials for that operation only. They expire automatically and are never stored.

## Setup requirements

Enterprise Connect requires the following:

* A dedicated instance provisioned for your Unstructured account, with Enterprise Connect enabled. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
* An AWS account where you can create and manage IAM roles.

## Configure your connector

Select your preferred interface (Pipelines or API) to view setup instructions for your connector.

<Note>
  A connector configured with Enterprise Connect is available only in the workspace where it was created. It is not accessible from other workspaces in your Unstructured account.
</Note>

| Connector      | Source                                                                                                                                                                            | Destination                                                                                                                                                                                 |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| AWS S3         | [Pipelines](/pipelines/sources/s3#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/sources/s3#set-up-enterprise-connect-authentication)                 | [Pipelines](/pipelines/destinations/s3#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/destinations/s3#set-up-enterprise-connect-authentication)                 |
| AWS OpenSearch | [Pipelines](/pipelines/sources/opensearch#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/sources/opensearch#set-up-enterprise-connect-authentication) | [Pipelines](/pipelines/destinations/opensearch#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/destinations/opensearch#set-up-enterprise-connect-authentication) |


# Onboarding in-VPC on your AWS platform
Source: https://docs.unstructured.io/business/aws/onboarding-in-vpc



<Note>
  The following information applies only to in-VPC deployments of [Unstructured Business](/business/overview).

  For dedicated instance deployments of Unstructured Business, contact your Unstructured sales representative,
  or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

  For dedicated instance deployments of Unstructured Business that also use PrivateLink, see
  the [onboarding process for AWS PrivateLink](/business/aws/aws-privatelink/onboarding-di-pl).
</Note>

After your organization has signed the **Business** account agreement with Unstructured, a member of the Unstructured technical enablement team will reach out to you to begin the
deployment onboarding process. To streamline this process, you are encouraged to begin setting up your target environment as soon as possible. Choose one of the following setup options:

* [Do it all for me](#do-it-all-for-me): Have Unstructured set up the required infrastructure in your AWS account and then deploy Unstructured Pipelines and the API into that newly created infrastructure.
* [Bring my own infrastructure](#bring-my-own-infrastructure): Set up the required infrastructure yourself in your AWS account, and then have Unstructured deploy Unstructured Pipelines and the API into your existing infrastructure.

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).

## Do it all for me

If you want Unstructured to set up the required infrastructure for you in your AWS account and then deploy Unstructured Pipelines and the API into that newly created infrastructure, then provide your Unstructured sales representative or technical enablement contact with
the access credentials for an IAM user or service principal in your AWS account that has the following required permissions.

### Core networking permissions

For VPC and subnet management:

* `ec2:CreateVpc`
* `ec2:CreateSubnet`
* `ec2:CreateRouteTable`
* `ec2:CreateInternetGateway`
* `ec2:CreateNatGateway`
* `ec2:ModifyVpcAttribute` (for DNS settings)
* `ec2:AssociateRouteTable`, `ec2:CreateRoute` (for public and private route tables)
* `ec2:AllocateAddress` (for Elastic IP assignment to the NAT Gateway)

For security group rules:

* `ec2:AuthorizeSecurityGroupIngress/Egress` (to configure cluster and node security groups to allow VPC CIDR traffic)

### EKS permissions

For the cluster role:

* Attach the managed policies `AmazonEKSClusterPolicy` and `AmazonEKSVPCResourceController` to a role with `sts:AssumeRole` trust for `eks.amazonaws.com`

For the node group role:

Attach these managed policies:

* `AmazonEKSWorkerNodePolicy` (for node operations)
* `AmazonEKS_CNI_Policy` (for networking)
* `AmazonEC2ContainerRegistryReadOnly` (for ECR access)

For OIDC integration:

* `iam:CreateOpenIDConnectProvider` (to associate the EKS cluster with IAM OIDC)
* `iam:CreateRole` + `iam:AttachRolePolicy` (for service accounts in the `recommender`, `etl-operator`, and `data-broker` namespaces)

### Storage and database

These permissions:

* `s3:CreateBucket`
* `s3:PutBucketVersioning`
* `s3:PutBucketEncryption`

For these S3 buckets:

* `u10d-*-etl-blob-cache`
* `u10d-*-etl-job-db`
* `u10d-*-etl-job-status`
* `u10d-*-job-files`

For RDS:

* `rds:CreateDBInstance`
* `rds:CreateDBSubnetGroup`
* `rds:CreateDBSecurityGroup` + `ec2:AuthorizeSecurityGroupIngress` (to allow VPC CIDR access)

### Add-ons and utilities

For the EBS CSI Driver:

* `eks:CreateAddon` with IAM role attachment permissions for the `ebs.csi.aws.com` service account

For the SSH Key:

* `ec2:CreateKeyPair` + `ec2:ExportKeyPair` (for node group remote access)

### Cross-service requirements

* For IAM: `iam:PassRole` (to assign roles to EKS, RDS, and S3)
* For KMS: `kms:CreateKey` (if using CMK for S3 and RDS encryption)
* For CloudFormation: `cloudformation:*`

For least privilege, scope resource ARNs in policies (for example, restrict S3 bucket names with wildcards such as `u10d-*-etl*`).
The EKS Pod Identity Agent requires `eks-auth:AssumeRoleForPodIdentity` permission on node roles when used with IRSA.

## Bring my own infrastructure

If you want to set up the required infrastructure yourself, set things up as follows within your AWS account for Unstructured to deploy Unstructured Pipelines and the API into.

You must also provide your Unstructured sales representative or technical enablement contact with
the access credentials for an IAM user or service principal in your AWS account that has access to the target Amazon Elastic Kubernetes Service (EKS) cluster to deploy the
Unstructured Pipelines and the API into.

### VPC and networking

* **VPC**

  * CIDR: `10.0.0.0/16` - Any CIDR should work, but make sure it has enough space.
  * DNS Hostnames: Enabled
  * DNS Support: Enabled

* **Internet Gateway**

  * Attached to the VPC

* **Public Subnet**

  * CIDR: `10.0.0.0/24`
  * Public IP on launch: true
  * Availability Zone: `${region}a`

* **NAT Gateway + Elastic IP**

  * Lives in the public subnet

* **Private Subnets (x2)**

  * CIDRs: `10.0.1.0/24`, `10.0.2.0/24`
  * AZs: `${region}a` and `${region}b`

* **Route Tables**

  * Public: default route (`0.0.0.0/0`) via IGW
  * Private (x2): default route via NAT Gateway

### **IAM roles and policies**

* **EKS Cluster Role**

  * Trusts: `eks.amazonaws.com`
  * Attached policies:

    * `AmazonEKSClusterPolicy`
    * `AmazonEKSVPCResourceController`

* **EKS Node Group Role**

  * Trusts: `ec2.amazonaws.com`, `eks.amazonaws.com`
  * Attached policies:

    * `AmazonEKSWorkerNodePolicy`
    * `AmazonEKS_CNI_Policy`
    * `AmazonEC2ContainerRegistryReadOnly`

* **OIDC Service Account IAM Roles (x3)**

  * Namespaces: `recommender`, `etl-operator`, `data-broker`
  * Each role assumes via `sts:AssumeRoleWithWebIdentity` with OIDC provider
  * Each has an S3 policy allowing access to specific buckets (see below)

### **EKS cluster**

* **EKS Control Plane**

  * Version: `1.31` or greater
  * Subnet: Private subnets only
  * Public endpoint access: Enabled
  * Private endpoint access: Disabled

* **Node Group**

  * Instance type: `c5.4xlarge` (or larger, depending on cost factors)
  * Disk size: 100 GB
  * Desired size: 2 (min 2, max 5)
  * Remote SSH access: Enabled (with generated SSH key)
  * SSH key: Key pair created and exported

* **Security Groups**

  * EKS Cluster SG (implicitly created by AWS)
  * Node SG: Allows all traffic within cluster CIDR (`10.0.0.0/16`), self, and metadata IP
  * Egress: Allows all

#### **Kubernetes add-ons**

Installed via `aws.eks.Addon`:

* **EKS Pod Identity Agent**

  * Version: `v1.3.4-eksbuild.1`

* **Metrics Server**

  * Version: `v0.7.2-eksbuild.1`

* **EBS CSI Driver**

  * Version: `v1.38.1-eksbuild.2`
  * Configured with:

    * Service account annotation: `eks.amazonaws.com/role-arn`
    * Pod identity access annotation

#### **Storage class**

* Name: `ebs-sc`
* Default: Yes
* Provisioner: `ebs.csi.aws.com`
* Parameters: `type=gp3`, `encrypted=true`
* Volume Binding Mode: `WaitForFirstConsumer`

### **RDS**

* **RDS Subnet Group**

  * Uses the private subnets

* **RDS Instance**

  * Engine: Postgres 16
  * Size: `db.t3.micro`
  * Allocated storage: 20 GB
  * Auth: Setup a Username and Password, keep secure.
  * Security group: Allows all traffic from `10.0.0.0/16` (keep in mind your CIDR group from the VPC)
  * DB name: `postgres`

### **S3 buckets**

* `u10d-{stack_name}-etl-blob-cache`
* `u10d-{stack_name}-etl-job-db`
* `u10d-{stack_name}-etl-job-status`
* `u10d-{stack_name}-job-files`

All created with:

* Versioning enabled
* Server-side encryption (AES256)
* Force destroy: true

### **Keys**

* **SSH Key Pair** (RSA 4096-bit)

  * Key exported as `private_key` (PEM)

### Secrets and ConfigMaps

After your infrastructure is set up, but before Unstructured can deploy Unstructured Pipelines and the API into your insfrastructure,
Unstructured will need to know the values of the following Secrets and ConfigMaps. These must be provided to Unstructured as a
set of YAML files in Kubernetes [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) and
[ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) format.

The Secrets are as follows.

#### **Blob storage credentials**

* `BLOB_STORAGE_ADAPTER_ACCESS_KEY_ID`
* `BLOB_STORAGE_ADAPTER_SECRET_ACCESS_KEY`
* `BLOB_STORAGE_ADAPTER_REGION_NAME`

#### **Database credentials**

* `DB_USERNAME`
* `DB_PASSWORD`
* `DB_HOST`
* `DB_NAME`
* `DB_DATABASE` (used in `platform-api` only)

#### **Authentication**

* `JWT_SECRET_KEY`
* `AUTH_STRATEGY` (sometimes encoded, sometimes not)
* `SESSION_SECRET`
* `SHARED_SECRET`
* `KEYCLOAK_CLIENT_SECRET`
* `KEYCLOAK_ADMIN_SECRET`
* `KEYCLOAK_ADMIN`
* `KEYCLOAK_ADMIN_PASSWORD`
* `API_BEARER_TOKEN`

The ConfigMaps are as follows.

#### **Blob storage settings**

* `BLOB_STORAGE_ADAPTER_TYPE` (always `s3` for AWS)
* `BLOB_STORAGE_ADAPTER_BUCKET`
* `ETL_BLOB_CACHE_BUCKET_NAME`
* `ETL_API_BLOB_STORAGE_ADAPTER_BUCKET`
* `ETL_API_BLOB_STORAGE_ADAPTER_TYPE`
* `ETL_API_DB_REMOTE_BUCKET_NAME`
* `ETL_API_JOB_STATUS_DEST_BUCKET_NAME`
* `JOB_STATUS_BUCKET_NAME`
* `JOB_DB_BUCKET_NAME`

#### **Environment**

* `ENV`
* `ENVIRONMENT`
* `JOB_ENV`
* `JOB_ENVIRONMENT`

#### **Observability and OpenTelemetry (OTel)**

* `JOB_OTEL_EXPORTER_OTLP_ENDPOINT`
* `JOB_OTEL_METRICS_EXPORTER`
* `JOB_OTEL_TRACES_EXPORTER`
* `OTEL_EXPORTER_OTLP_ENDPOINT`
* `OTEL_METRICS_EXPORTER`
* `OTEL_TRACES_EXPORTER`

#### **Unstructured API and authentication**

* `UNSTRUCTURED_API_URL`
* `JWKS_URL`
* `JWT_ISSUER`
* `JWT_AUDIENCE`
* `SINGLE_PLANE_DEPLOYMENT`

#### **Front end and dashboard**

* `API_BASE_URL`
* `API_CLIENT_BASE_URL`
* `API_URL`
* `APM_SERVICE_NAME`
* `APM_SERVICE_NAME_CLIENT`
* `AUTH_STRATEGY`
* `FRONTEND_BASE_URL`
* `KEYCLOAK_CALLBACK_URL`
* `KEYCLOAK_CLIENT_ID`
* `KEYCLOAK_DOMAIN`
* `KEYCLOAK_REALM`
* `KEYCLOAK_SSL_ENABLED`
* `KEYCLOAK_TRUST_ISSUER`
* `PUBLIC_BASE_URL`
* `PUBLIC_RELEASE_CHANNEL`

#### **Sentry and feature flags**

* `SENTRY_DSN`
* `SENTRY_SAMPLE_RATE`
* `WORKFLOW_NODE_EDITOR_FF_REQUEST_FORM`
* `CUSTOM_WORKFLOW_FF_REQUEST_FORM`

#### **Redis**

* `REDIS_DSN`

#### **Other**

* `IMAGE_PULL_SECRETS`
* `PRIVATE_KEY_SECRETS_ADAPTER_TYPE`
* `PRIVATE_KEY_SECRETS_ADAPTER_AWS_REGION`
* `SECRETS_ADAPTER_TYPE`
* `SECRETS_ADAPTER_AWS_REGION`

The preceding Secrets and ConfigMaps must be added to the following files:

| File name                             | Type      | Resource name              | Namespace      | Data keys                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------------- | --------- | -------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data-broker-env-cm.yaml`             | ConfigMap | `data-broker-env`          | `api`          | `JOB_STATUS_BUCKET_NAME`, `JOB_DB_BUCKET_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `data-broker-env-secret.yaml`         | Secret    | `data-broker-env`          | `api`          | ` BLOB_STORAGE_ADAPTER_ACCESS_KEY_ID`, `BLOB_STORAGE_ADAPTER_REGION_NAME`, `BLOB_STORAGE_ADAPTER_SECRET_ACCESS_KEY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `dataplane-api-env-cm.yaml`           | Secret    | `dataplane-api-env`        | `api`          | ` DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `etl-operator-env-cm.yaml`            | ConfigMap | `etl-operator-env`         | `etl-operator` | ` BLOB_STORAGE_ADAPTER_BUCKET`, `JOB_STATUS_BUCKET_NAME`, `JOB_DB_BUCKET_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`, `ENV`, `ENVIRONMENT`, `REDIS_DSN`, `ETL_API_BLOB_STORAGE_ADAPTER_BUCKET`, `ETL_API_BLOB_STORAGE_ADAPTER_TYPE`, `ETL_API_DB_REMOTE_BUCKET_NAME`, `ETL_API_JOB_STATUS_DEST_BUCKET_NAME (x2)`, `ETL_BLOB_CACHE_BUCKET_NAME`, `IMAGE_PULL_SECRETS`, `JOB_ENV`, `JOB_ENVIRONMENT`, `JOB_OTEL_EXPORTER_OTLP_ENDPOINT`, `JOB_OTEL_METRICS_EXPORTER`, `JOB_OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_METRICS_EXPORTER`, `OTEL_TRACES_EXPORTER`, `UNSTRUCTURED_API_URL` |
| `etl-operator-env-secret.yaml`        | Secret    | `etl-operator-env`         | `etl-operator` | ` BLOB_STORAGE_ADAPTER_ACCESS_KEY_ID`, `BLOB_STORAGE_ADAPTER_REGION_NAME`, `BLOB_STORAGE_ADAPTER_SECRET_ACCESS_KEY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `frontend-env-cm.yaml`                | ConfigMap | `frontend-env`             | `www`          | `API_BASE_URL`, `API_CLIENT_BASE_URL`, `API_URL`, `APM_SERVICE_NAME`, `APM_SERVICE_NAME_CLIENT`, `AUTH_STRATEGY`, `ENV`, `FRONTEND_BASE_URL`, `KEYCLOAK_CALLBACK_URL`, `KEYCLOAK_CLIENT_ID`, `KEYCLOAK_DOMAIN`, `KEYCLOAK_REALM`, `KEYCLOAK_SSL_ENABLED`, `KEYCLOAK_TRUST_ISSUER`, `PUBLIC_BASE_URL`, `PUBLIC_RELEASE_CHANNEL`, `SENTRY_DSN`, `SENTRY_SAMPLE_RATE`, `WORKFLOW_NODE_EDITOR_FF_REQUEST_FORM`, `CUSTOM_WORKFLOW_FF_REQUEST_FORM`                                                                                                                                                 |
| `frontend-env-secret.yaml`            | Secret    | `frontend-env`             | `www`          | `API_BEARER_TOKEN`, `KEYCLOAK_ADMIN_SECRET`, `KEYCLOAK_CLIENT_SECRET`, `SESSION_SECRET`, `SHARED_SECRET`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `keycloak-secret.yaml`                | Secret    | `phasetwo-keycloak-env`    | `www`          | `KEYCLOAK_ADMIN`, `KEYCLOAK_ADMIN_PASSWORD`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `platform-api-env-cm.yaml`            | ConfigMap | `platform-api-env`         | `api`          | `JWKS_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `SINGLE_PLANE_DEPLOYMENT`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `platform-api-env-secret.yaml`        | Secret    | `platform-api-env`         | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`, `DB_DATABASE`, `JWT_SECRET_KEY`, `AUTH_STRATEGY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `recommender-env-cm.yaml`             | ConfigMap | `recommender-env`          | `recommender`  | `BLOB_STORAGE_ADAPTER_TYPE`, `ETL_BLOB_CACHE_BUCKET_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `recommender-env-secret.yaml`         | Secret    | `recommender-env`          | `recommender`  | `BLOB_STORAGE_ADAPTER_ACCESS_KEY_ID`, `BLOB_STORAGE_ADAPTER_REGION_NAME`, `BLOB_STORAGE_ADAPTER_SECRET_ACCESS_KEY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `secret-provider-api-env-cm.yaml`     | ConfigMap | `secrets-provider-api-env` | `secrets`      | `ENV`, `ENVIRONMENT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_METRICS_EXPORTER`, `OTEL_TRACES_EXPORTER`, `PRIVATE_KEY_SECRETS_ADAPTER_AWS_REGION`, `PRIVATE_KEY_SECRETS_ADAPTER_TYPE`, `SECRETS_ADAPTER_AWS_REGION`, `SECRETS_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                      |
| `secret-provider-api-env-secret.yaml` | Secret    | `secrets-provider-api-env` | `secrets`      | `BLOB_STORAGE_ADAPTER_ACCESS_KEY_ID`, `BLOB_STORAGE_ADAPTER_REGION_NAME`, `BLOB_STORAGE_ADAPTER_SECRET_ACCESS_KEY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `usage-collector-env-secret.yaml`     | Secret    | `usage-collector-env`      | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

For example, for the `etl-operator-env-cm.yaml` [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) file, the contents would look like this:

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: data-broker-env
  namespace: api
data:
  JOB_STATUS_BUCKET_NAME: "<your-value>"
  JOB_DB_BUCKET_NAME: "<your-value>"
  BLOB_STORAGE_ADAPTER_TYPE: "<your-value>"
```

For the `etl-operator-env-secret.yaml` [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) file, the contents would look like this:

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: data-broker-env
  namespace: api
type: Opaque
stringData:
  BLOB_STORAGE_ADAPTER_ACCESS_KEY_ID: "<your-value>"
  BLOB_STORAGE_ADAPTER_REGION_NAME: "<your-value>"
  BLOB_STORAGE_ADAPTER_SECRET_ACCESS_KEY: "<your-value>"
```


# Unstructured Business on Amazon Web Services (AWS)
Source: https://docs.unstructured.io/business/aws/overview



## Getting started

To get started with a **Business** account deployment, your organization must first sign a **Business** account agreement with Unstructured. This agreement outlines the terms and conditions for your organization's use of Unstructured.

To begin the **Business** account agreement process, contact your Unstructured sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

After your organization signs the **Business** account agreement, a member of the Unstructured technical enablement team will reach out to begin the deployment onboarding process.

While you wait, you can get started by following the onboarding checklist for your deployment type, if it applies to you:

* For in-VPC deployments, follow [onboarding in-VPC on AWS](/business/aws/onboarding-in-vpc).

* For dedicated instance deployments with AWS PrivateLink, follow [onboarding a dedicated instance](/business/aws/aws-privatelink/onboarding-di-pl).

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Connecting to customer-managed services on Azure
Source: https://docs.unstructured.io/business/azure/azure-privatelink/connect-to-customer-managed-services



This page covers how to establish private connectivity between your dedicated instance and services running in your Azure environment that you self-host, such as MongoDB, Elasticsearch, or applications behind Azure API Management in internal VNet mode.

For Azure-managed services that support Private Endpoint access natively (Blob Storage, Azure OpenAI, AI Search, Azure SQL, Cosmos DB, PostgreSQL, MySQL, Synapse, Azure Databricks), see [Connecting to Azure-managed services](/business/azure/azure-privatelink/connect-to-services).

## How it works

Customer-managed services on Azure require you to front them with an **Azure Standard Load Balancer** and expose them via a **Private Link Service**. Unstructured then creates a Private Endpoint in the Unstructured-managed environment to connect to your service privately.

In Azure terminology, you are the *service provider* and Unstructured is the *service consumer*.

## Information exchange

The table below uses **Order** to indicate the general sequence of the information exchange — items with the same Order value can usually be provided at the same stage. The `<DI_AZURE_SUBSCRIPTION_ID>` value is a placeholder; contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/) to get the actual value for your dedicated instance.

<Note>
  Your service must be in the same Azure region as your dedicated instance — cross-region Private Link is not supported. The example region code in the table below is illustrative; use the region your dedicated instance is deployed in. See [Known limitations](/business/dedicated-instances/requirements#known-limitations).
</Note>

| Order | Information Required               | Description                                                                    | Example                                                                               | Provided by  |
| ----- | ---------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | ------------ |
| 1     | Service Type                       | The service being accessed                                                     | `MongoDB`, `Elasticsearch`, `Internal API`                                            | Customer     |
| 1     | Service Region                     | Azure region where the service is hosted                                       | `eastus`                                                                              | Customer     |
| 1     | Service Port                       | Port(s) the service listens on                                                 | `27017` (MongoDB), `9200` (Elasticsearch), `443`                                      | Customer     |
| 2     | Unstructured Azure Subscription ID | Subscription ID to add to the visibility settings of your Private Link Service | `<DI_AZURE_SUBSCRIPTION_ID>`                                                          | Unstructured |
| 3     | Private Link Service Resource ID   | Full resource ID of your Private Link Service                                  | `/subscriptions/.../providers/Microsoft.Network/privateLinkServices/customer-app-pls` | Customer     |
| 3     | Service Endpoint                   | The endpoint hostname or IP for connection                                     | Custom DNS or Private Link Service alias                                              | Customer     |

## Steps

### Step 1: Provide service information

Provide Unstructured with the service type, region, and port via support ticket. Unstructured will share the Unstructured Azure Subscription ID you need to complete Step 3.

### Step 2: Create a Standard Load Balancer and Private Link Service

Deploy an Azure Standard Load Balancer in front of your service, then create a Private Link Service attached to the load balancer's frontend IP configuration. [Learn how](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview).

**Using the Azure CLI:**

```bash theme={null}
az network private-link-service create \
  --name "customer-app-pls" \
  --resource-group "<RESOURCE_GROUP>" \
  --vnet-name "<VNET_NAME>" \
  --subnet "<SUBNET_NAME>" \
  --lb-name "<LOAD_BALANCER_NAME>" \
  --lb-frontend-ip-configs "<FRONTEND_IP_CONFIG_NAME>" \
  --location "<REGION>"
```

Replace:

* `<RESOURCE_GROUP>` — Resource group for the Private Link Service and related resources
* `<VNET_NAME>` — Virtual network name
* `<SUBNET_NAME>` — Subnet used for the Private Link Service (must meet Azure subnet requirements for Private Link)
* `<LOAD_BALANCER_NAME>` — Name of the Standard Load Balancer fronting your application
* `<FRONTEND_IP_CONFIG_NAME>` — Frontend IP configuration name on the load balancer
* `<REGION>` — Azure region (for example `eastus`)

### Step 3: Add Unstructured's subscription to your Private Link Service visibility

Update the visibility settings on your Private Link Service to include Unstructured's Azure Subscription ID so Unstructured can create a Private Endpoint connecting to it. [Learn how](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#control-service-exposure).

### Step 4: Share your Private Link Service details with Unstructured

Provide Unstructured with your Private Link Service Resource ID and service endpoint via support ticket.

Unstructured will create a Private Endpoint in the Unstructured-managed environment targeting your service.

### Step 5: Approve Unstructured's connection request

Once Unstructured creates the Private Endpoint, a connection request appears on your Private Link Service. Approve it as soon as possible to avoid delays. [Learn how](https://learn.microsoft.com/en-us/azure/private-link/manage-private-endpoint).

**Using the Azure CLI:**

```bash theme={null}
az network private-link-service connection approve \
  --resource-group "<RESOURCE_GROUP>" \
  --service-name "customer-app-pls" \
  --name "<CONNECTION_NAME>"
```

Replace `<CONNECTION_NAME>` with the name of the pending connection request (shown in the Azure portal or provided by Unstructured).

## Private API Gateway use case

If your application uses Azure API Management in private-only (internal VNet) mode, you cannot target the API Management resource directly with a Private Link Service. Deploy an Azure Standard Load Balancer in front of API Management and create a Private Link Service that points to the load balancer.

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Connecting to Azure-managed services
Source: https://docs.unstructured.io/business/azure/azure-privatelink/connect-to-services



This page describes how to establish private connectivity between your dedicated instance and Azure-managed services. For self-hosted applications, see [Connecting to customer-managed services on Azure](/business/azure/azure-privatelink/connect-to-customer-managed-services).

## Azure managed services

This section covers Azure-managed services that have service-specific private connectivity requirements.

The tables below use **Order** to indicate the general sequence of the information exchange — items with the same Order value can usually be provided at the same stage. Values like `<UNSTRUCTURED_SP_ID>` and `<DI_AZURE_SUBSCRIPTION_ID>` are placeholders; contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/) to get the actual values for your dedicated instance.

<Note>
  Each Azure service you connect to via Private Link must be in the same Azure region as your dedicated instance — cross-region Private Link is not supported. The example region codes in the tables below are illustrative; use the region your dedicated instance is deployed in. See [Known limitations](/business/dedicated-instances/requirements#known-limitations).
</Note>

### Azure Blob Storage

This applies to Azure Blob Storage and Azure Data Lake Storage Gen2 (ADLS Gen2).

| Order | Information Required              | Description                                             | Example                                                                            | Owner        |
| ----- | --------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------ |
| 1     | Storage Account Name              | Name of the Azure Storage Account                       | `customerstorageacct`                                                              | Customer     |
| 1     | Storage Account Resource ID       | Full Azure Resource ID of the storage account           | `/subscriptions/…/providers/Microsoft.Storage/storageAccounts/customerstorageacct` | Customer     |
| 1     | Container Name(s)                 | Containers Unstructured needs to access                 | `documents, processed-data`                                                        | Customer     |
| 1     | Storage Account Region            | Region where storage account is located                 | `eastus`                                                                           | Customer     |
| 1     | Sub-resource Type                 | `blob`, `dfs` (for ADLS Gen2), `file`, `table`, `queue` | `blob`, `dfs`                                                                      | Customer     |
| 2     | Unstructured Service Principal ID | Service Principal ID for RBAC role assignment           | `<UNSTRUCTURED_SP_ID>`                                                             | Unstructured |
| 2     | Unstructured Subscription ID      | Subscription ID for private endpoint approval           | `<DI_AZURE_SUBSCRIPTION_ID>`                                                       | Unstructured |

For ADLS Gen2, use sub-resource type `dfs` for hierarchical namespace operations. You must approve the private endpoint connection request.  To do so, navigate to **Storage Account > Networking > Private endpoint connections**.

**Example: Azure role assignment**

The following Azure CLI command assigns the **Storage Blob Data Reader** role to Unstructured’s service principal so the platform can read blobs in your storage account.

```bash theme={null}
az role assignment create \
  --assignee "<UNSTRUCTURED_SP_ID>" \
  --role "Storage Blob Data Reader" \
  --scope "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT>"
```

For **read/write** access, use the **Storage Blob Data Contributor** role instead of **Storage Blob Data Reader**.

Replace:

* `<UNSTRUCTURED_SP_ID>` — Service principal ID for Unstructured (provided during onboarding)
* `<SUBSCRIPTION_ID>` — Your Azure subscription ID
* `<RESOURCE_GROUP>` — Resource group that contains the storage account
* `<STORAGE_ACCOUNT>` — Storage account name

***

### Azure OpenAI

Azure OpenAI supports private endpoint access. Unstructured creates a private endpoint in our VNet to access the your Azure OpenAI resource privately.

| Order | Information Required              | Description                                   | Example                                                                           | Owner        |
| ----- | --------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------- | ------------ |
| 1     | Azure OpenAI Resource Name        | Name of the Azure OpenAI resource             | `customer-openai`                                                                 | Customer     |
| 1     | Azure OpenAI Resource ID          | Full Azure Resource ID                        | `/subscriptions/…/providers/Microsoft.CognitiveServices/accounts/customer-openai` | Customer     |
| 1     | Azure OpenAI Region               | Region where the resource is deployed         | `eastus`                                                                          | Customer     |
| 1     | Deployment Name(s)                | Model deployment names to access              | `gpt-4-deployment, embedding-deployment`                                          | Customer     |
| 1     | Sub-resource Type                 | Always `account` for Azure OpenAI             | `account`                                                                         | Customer     |
| 2     | Unstructured Service Principal ID | Service Principal ID for RBAC role assignment | `<UNSTRUCTURED_SP_ID>`                                                            | Unstructured |
| 2     | Unstructured Subscription ID      | Subscription ID for private endpoint approval | `<DI_AZURE_SUBSCRIPTION_ID>`                                                      | Unstructured |

**Example: Azure role assignment for OpenAI**

Use Azure CLI to assign the **Cognitive Services OpenAI User** role to Unstructured’s service principal for your Azure OpenAI resource:

```bash theme={null}
az role assignment create \
  --assignee "<UNSTRUCTURED_SP_ID>" \
  --role "Cognitive Services OpenAI User" \
  --scope "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.CognitiveServices/accounts/<OPENAI_RESOURCE>"
```

Replace:

* `<UNSTRUCTURED_SP_ID>` — Service principal ID for Unstructured (provided during onboarding)
* `<SUBSCRIPTION_ID>` — Your Azure subscription ID
* `<RESOURCE_GROUP>` — Resource group that contains the storage account
* `<OPENAI_RESOURCE>` — Your Azure OpenAI Resource ID

After the role assignment:

1. **Firewall:** On the Azure OpenAI resource, configure networking so **public access is disabled** and traffic is allowed only as required for your private endpoint design.
2. **Approve the connection:** In the Azure portal, go to the OpenAI resource > **Networking** > **Private endpoint connections**, then approve the pending private endpoint connection from Unstructured.

***

### Azure Databricks

Azure Databricks supports Private Link for both front-end (user-to-workspace) and back-end (compute-to-control plane) connectivity. You must have Private Link enabled on your workspace.

| Order | Information Required             | Description                                   | Example                                                                         | Owner        |
| ----- | -------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------- | ------------ |
| 1     | Databricks Workspace URL         | The workspace URL                             | `adb-1234567890123456.12.azuredatabricks.net`                                   | Customer     |
| 1     | Databricks Workspace Resource ID | Full Azure Resource ID of the workspace       | `/subscriptions/…/providers/Microsoft.Databricks/workspaces/customer-workspace` | Customer     |
| 1     | Databricks Workspace Region      | Azure region where workspace is deployed      | `eastus`                                                                        | Customer     |
| 1     | Sub-resource Type                | `databricks_ui_api` for workspace access      | `databricks_ui_api`                                                             | Customer     |
| 2     | Unstructured Subscription ID     | Subscription ID for private endpoint approval | `<DI_AZURE_SUBSCRIPTION_ID>`                                                    | Unstructured |
| 3     | Private endpoint DNS             | The private endpoint DNS for the workspace    | `adb-1234567890123456.12.privatelink.azuredatabricks.net`                       | Customer     |

**Example: Approve the Databricks private endpoint connection**

After Unstructured creates the private endpoint toward your workspace, approve the pending connection using Azure CLI:

```bash theme={null}
az network private-endpoint-connection approve \
  --resource-group "<RESOURCE_GROUP>" \
  --resource-name "<WORKSPACE_NAME>" \
  --name "<PRIVATE_ENDPOINT_NAME>" \
  --type Microsoft.Databricks/workspaces
```

You can also approve the same request in the **Azure portal**: open the Databricks workspace resource > **Networking** > **Private endpoint connections**, then approve the pending connection from Unstructured.

Replace:

* `<RESOURCE_GROUP>` — Resource group that contains the Databricks workspace
* `<WORKSPACE_NAME>` — Name of the Databricks workspace resource
* `<PRIVATE_ENDPOINT_NAME>` — Name of the private endpoint connection (shown in the portal or provided by Unstructured)

***

## Azure Managed Services with private endpoint access

This section applies to Azure Managed Services that support private endpoint access, including: **Azure AI Search, Azure SQL Database, Azure SQL Managed Instance, Azure Cosmos DB, Azure Database for PostgreSQL, Azure Database for MySQL,** and **Azure Synapse Analytics**.

<Note>
  The **Order** column indicates the general sequence for the information exchange. Items with the same order value can usually be provided at the same stage.
</Note>

| Order | Information Required              | Description                                              | Example                                                                | Owner        |
| ----- | --------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------- | ------------ |
| 1     | Service Type                      | The Azure managed service being accessed                 | `Azure AI Search`, `Azure SQL`                                         | Customer     |
| 1     | Resource Name                     | Name of the Azure resource                               | `my-search-service`                                                    | Customer     |
| 1     | Resource ID                       | Full Azure Resource ID                                   | `/subscriptions/…/providers/Microsoft.Search/searchServices/my-search` | Customer     |
| 1     | Service Region                    | Region where the service is hosted                       | `eastus`                                                               | Customer     |
| 1     | Sub-Resource Type                 | The sub-resource type for Private Link (see table below) | `searchService`, `sqlServer`                                           | Customer     |
| 1     | Service Port                      | Port the service listens on                              | `443` (AI Search), `1433` (SQL)                                        | Customer     |
| 2     | Unstructured Service Principal ID | Service Principal ID for RBAC role assignment            | `<UNSTRUCTURED_SP_ID>`                                                 | Unstructured |
| 2     | Unstructured Subscription ID      | Subscription ID to approve for private endpoint          | `<DI_AZURE_SUBSCRIPTION_ID>`                                           | Unstructured |

**Sub-resource types by service:**

| Service                       | Sub-Resource Type           |
| ----------------------------- | --------------------------- |
| AI Search                     | `searchService`             |
| Azure SQL Database            | `sqlServer`                 |
| Azure SQL Managed Instance    | `managedInstance`           |
| Azure Cosmos DB (SQL API)     | `Sql`                       |
| Azure Cosmos DB (MongoDB API) | `MongoDB`                   |
| Azure Database for PostgreSQL | `postgresqlServer`          |
| Azure Database for MySQL      | `mysqlServer`               |
| Azure Synapse Analytics       | `Sql`, `SqlOnDemand`, `Dev` |


# Connector support for Azure Private Link
Source: https://docs.unstructured.io/business/azure/azure-privatelink/connector-support



The following table summarizes support and configuration for each connector in a dedicated instance deployment.

### Azure Connector Coverage

| Connector                                                | Private Link supported | Approach                                                               | Configuration information                                                                                         |
| -------------------------------------------------------- | :--------------------: | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Customer → Unstructured (Pipelines/API)**              |            ✅           | Private endpoint                                                       | [Link](/business/azure/azure-privatelink/onboarding-di-pl#step-3-connect-to-the-unstructured-platform)            |
| **Azure Blob Storage / ADLS Gen2**                       |            ✅           | Private endpoint                                                       | [Link](/business/azure/azure-privatelink/connect-to-services#azure-blob-storage)                                  |
| **Azure OpenAI**                                         |            ✅           | Private endpoint                                                       | [Link](/business/azure/azure-privatelink/connect-to-services#azure-openai)                                        |
| **Azure OpenAI via API Management**                      |            ✅           | Private endpoint + APIM Private mode                                   | Contact Unstructured                                                                                              |
| **Azure AI Search**                                      |            ✅           | Private endpoint                                                       | [Link](/business/azure/azure-privatelink/connect-to-services#azure-managed-services-with-private-endpoint-access) |
| **Databricks (Azure)**                                   |            ✅           | Private Link                                                           | [Link](/business/azure/azure-privatelink/connect-to-services#azure-databricks)                                    |
| **Azure SQL / Cosmos DB / PostgreSQL / MySQL / Synapse** |            ✅           | Private endpoint                                                       | [Link](/business/azure/azure-privatelink/connect-to-services#azure-managed-services-with-private-endpoint-access) |
| **Customer Managed Services (Azure)**                    |            ✅           | Private Link Service + Standard Load Balancer                          | [Link](/business/azure/azure-privatelink/connect-to-customer-managed-services)                                    |
| **OneDrive / SharePoint / Outlook**                      |            ❌           | Microsoft SaaS — requires public egress                                | Requires [public egress](/business/dedicated-instances/security/security-model#network-access-controls)           |
| **Google Cloud Storage**                                 |            ❌           | Cross-cloud access is not supported through Azure private connectivity | Requires [public egress](/business/dedicated-instances/security/security-model#network-access-controls)           |


# Configure DNS for Azure Private Link
Source: https://docs.unstructured.io/business/azure/azure-privatelink/dns-privatelink



This page describes how to configure DNS so your dedicated instance hostname (`<companyname>.privatelink.unstructuredapp.io`) resolves to the private IP of your Azure Private Endpoint. This step happens after you create your Private Endpoint as part of [onboarding](/business/azure/azure-privatelink/onboarding-di-pl).

Clients must connect using the dedicated instance hostname because TLS is terminated using a provider-managed certificate. The two most common setups are below — choose the one that matches your authoritative DNS.

## Choose your setup

| Your authoritative DNS is...                                           | Go to                                         |
| ---------------------------------------------------------------------- | --------------------------------------------- |
| Azure DNS (Azure Private DNS Zones)                                    | [You use Azure DNS](#you-use-azure-dns)       |
| An internal DNS system such as Infoblox, BIND, or Active Directory DNS | [You use your own DNS](#you-use-your-own-dns) |

<Note>
  A Zero Trust Network (such as Zscaler) is a network path, not a DNS authority. Clients connecting through a ZTN still resolve DNS through whichever DNS system you use, so the section that matches your authoritative DNS already covers them. Note that some ZTN configurations intercept or proxy DNS queries — if your ZTN performs DNS interception, verify that queries for `<companyname>.privatelink.unstructuredapp.io` are forwarded to your configured DNS system and not resolved independently by the ZTN.
</Note>

## You use Azure DNS

This is the simplest setup if you already use Azure DNS in your subscription.

### In-VNet clients

When you create your Private Endpoint, select **Integrate with private DNS zone** and choose or create an Azure Private DNS Zone named `privatelink.unstructuredapp.io`. Azure automatically creates an A record in the zone that resolves your dedicated instance hostname to the Private Endpoint's private IP. Link the zone to every VNet that needs access. No additional DNS configuration is needed.

For more on automatic registration, see [Private Endpoint DNS integration scenarios](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns-integration).

### On-premises clients (ExpressRoute or VPN)

The Private DNS Zone integration in the previous section only works for clients inside a linked VNet. If you have on-premises clients reaching the platform over ExpressRoute or a VPN, bridge your on-premises DNS to Azure DNS with an [Azure DNS Private Resolver](https://learn.microsoft.com/en-us/azure/dns/dns-private-resolver-overview) inbound endpoint and a conditional forwarder.

This assumes you already have a private network connection (ExpressRoute or VPN) between your on-premises network and your VNet, and that the Private DNS Zone is linked to the VNet that hosts the Private Resolver.

1. **Create an Azure DNS Private Resolver inbound endpoint.** In the Azure portal, go to **DNS Private Resolvers**, create a resolver in the VNet linked to the Private DNS Zone, then add an **inbound endpoint** in a dedicated subnet. Azure assigns the endpoint a private IP (for example, `10.0.10.4`). Note this IP.

2. **Add a conditional forwarder on your on-premises DNS.** Forward queries for `privatelink.unstructuredapp.io` to the inbound endpoint IP. Forwarding the parent zone covers `<companyname>.privatelink.unstructuredapp.io` and all subdomains (for example `api.<companyname>.privatelink.unstructuredapp.io`) automatically — no per-instance rules are needed.

   Examples for common DNS systems:

   * **Windows DNS Server / Active Directory DNS:** Open **DNS Manager → Conditional Forwarders → New Conditional Forwarder**. Enter `privatelink.unstructuredapp.io` as the DNS Domain and add the inbound endpoint IP.

   * **BIND:** Add the following to `named.conf`, then run `rndc reload`:

     ```text theme={null}
     zone "privatelink.unstructuredapp.io" {
         type forward;
         forward only;
         forwarders { 10.0.10.4; };
     };
     ```

   * **Infoblox:** Go to **Data Management → DNS → Forward Zones → Add**. Enter `privatelink.unstructuredapp.io` as the zone name and add the inbound endpoint IP as the forwarder.

3. **Validate** from an on-premises client. See [Validation](#validation).

## You use your own DNS

If your authoritative DNS is something other than Azure DNS — for example, Infoblox, BIND, or Active Directory DNS — this is the recommended path. The same records serve clients in your VNet, on-premises clients reaching the platform over ExpressRoute or VPN, and laptops routed through a Zero Trust Network. There is no separate setup per client location, and no Azure DNS Private Resolver is required.

Unstructured cannot create these records for you because your Private Endpoint and its private IP live in your subscription.

<Warning>
  **VNet workloads must use your internal DNS.** This setup only works for in-VNet clients if your VNet is configured with custom DNS servers that point to your internal DNS, instead of using Azure-provided DNS (`168.63.129.16`). Set the VNet's **DNS servers** to your internal DNS server IPs. Without this change, VMs and other in-VNet workloads will use the Azure-provided resolver, which has no knowledge of your internal records and will not resolve the hostname correctly.

  If your internal DNS servers need to resolve other Azure-internal hostnames, they must forward unresolved queries to `168.63.129.16`. Otherwise, Azure-internal resolution will break for those workloads.
</Warning>

### Recommended: A records to the Private Endpoint's private IP

Add an apex and wildcard A record on a per-dedicated-instance zone in your internal DNS. The records resolve directly to the Private Endpoint's private IP. The wildcard ensures all current and future subdomains — for example `api.` and `platform-api.` — resolve without additional DNS changes.

Example records for a Private Endpoint with private IP `10.0.5.12`:

| Record                                           | Type | Target      |
| ------------------------------------------------ | ---- | ----------- |
| `<companyname>.privatelink.unstructuredapp.io`   | A    | `10.0.5.12` |
| `*.<companyname>.privatelink.unstructuredapp.io` | A    | `10.0.5.12` |

An Azure Private Endpoint has a single primary private IP per region. A TTL of 300 seconds is recommended. Find the Private Endpoint's private IP in the Azure portal: open the Private Endpoint resource, go to **DNS configuration**, and read the **IP addresses** column.

### Multiple dedicated instances

Customers with multiple dedicated instances configure a separate zone per dedicated instance. For example, a customer with development, staging, and production instances configures:

* `<companyname>-dev.privatelink.unstructuredapp.io` → A records pointing at the dev instance's Private Endpoint IP
* `<companyname>-staging.privatelink.unstructuredapp.io` → A records pointing at the staging instance's Private Endpoint IP
* `<companyname>-prod.privatelink.unstructuredapp.io` → A records pointing at the prod instance's Private Endpoint IP

This avoids claiming the entire `privatelink.unstructuredapp.io` namespace on your internal DNS and lets you connect to additional dedicated instances later without conflicts.

## Validation

From within the target network, run the following:

```bash theme={null}
dig <companyname>.privatelink.unstructuredapp.io
dig api.<companyname>.privatelink.unstructuredapp.io
```

On Windows, use `nslookup` instead of `dig`.

Both queries should return private IP addresses within your VNet CIDR range. If they return public IP addresses or fail to resolve, see [Troubleshooting](#troubleshooting).

## Troubleshooting

| Problem                                                                         | Likely cause                                                                                | What to check                                                                                                                                                                                                                               |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dig` returns public IP addresses                                               | The client is querying public DNS instead of the DNS system you configured                  | Confirm the client's resolver settings; confirm your internal DNS is authoritative for `privatelink.unstructuredapp.io` (own-DNS setup) or that your conditional forwarder is in place (Azure DNS + on-premises setup)                      |
| `dig` returns NXDOMAIN                                                          | The records are missing, or the client cannot reach your DNS                                | Confirm the records exist in your DNS system and that the client can reach your DNS resolvers                                                                                                                                               |
| Resolution works in the VNet but fails from on-premises (Azure DNS setup)       | The Private Resolver inbound endpoint or on-premises conditional forwarder is misconfigured | Confirm the inbound endpoint is healthy, that the VNet hosting the resolver is linked to the Private DNS Zone, and that the conditional forwarder targets the parent zone `privatelink.unstructuredapp.io` (not the full instance hostname) |
| Resolution works from on-premises but fails for in-VNet clients (own-DNS setup) | VNet workloads are using Azure-provided DNS instead of your internal DNS                    | Update the VNet's **DNS servers** setting to point to your internal DNS server IPs                                                                                                                                                          |
| Other Azure-internal hostnames fail to resolve after switching to custom DNS    | Your internal DNS server is not forwarding unresolved queries to `168.63.129.16`            | Configure your internal DNS to forward unresolved queries to `168.63.129.16`                                                                                                                                                                |
| TLS handshake fails after DNS resolves to a private IP                          | The client is connecting to a hostname not covered by the provider-managed certificate      | Confirm the client is connecting to a hostname under `<companyname>.privatelink.unstructuredapp.io`                                                                                                                                         |

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Onboarding Dedicated Instance + Azure Private Link
Source: https://docs.unstructured.io/business/azure/azure-privatelink/onboarding-di-pl



<Note>
  The following information applies only to dedicated instance deployments with Azure Private Link for [Unstructured Business](/business/overview) on Azure.

  For dedicated instance deployments of Unstructured Business to Azure *without* Azure Private Link, contact your Unstructured sales representative,
  or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

After your organization signs the **Business** account agreement with Unstructured, a member of the Unstructured technical enablement team will reach out to begin the deployment onboarding process.

## High-level onboarding process

| Step | Owner                   | Description                                                                             |
| :--: | ----------------------- | --------------------------------------------------------------------------------------- |
|   1  | Customer                | Provide Unstructured with the information required to provision your dedicated instance |
|   2  | Unstructured            | Build and configure your dedicated instance                                             |
|   3  | Customer                | Establish private connectivity to access Unstructured Pipelines and the API             |
|   4  | Customer + Unstructured | Connect your dedicated instance to your data sources and destinations                   |

<Note>
  Use a support ticket for all configuration data exchanges. Open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/) with the subject line **`PrivateLink Configuration Request — [Your Company Name]`**.
</Note>

## Step 1: Provide provisioning information

Provide the information listed in [Before you begin](/business/dedicated-instances/before-you-begin#what-to-prepare-before-onboarding) to Unstructured to begin provisioning your dedicated instance.

## Step 2: Unstructured provisions your dedicated instance

No action is required from you during this step. Once provisioning is complete, Unstructured will provide you with:

* The Azure Subscription ID your dedicated instance is hosted in
* The Private Link Service Resource ID for your dedicated instance

## Step 3: Connect to the Unstructured Platform

Once your dedicated instance is provisioned, establish private connectivity from your environment so your users can access Unstructured Pipelines and the API.

In Azure terminology, Unstructured is the *service provider* (publishing a Private Link Service) and you are the *service consumer* (creating a Private Endpoint to connect to it).

### Information exchange

| Information                       | Description                                                  | Example                                                                      | Provided by  |
| --------------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------- | ------------ |
| Private Link Service Resource ID  | Resource ID of the Unstructured Private Link Service         | `/subscriptions/.../providers/Microsoft.Network/privateLinkServices/uio-pls` | Unstructured |
| Service Region                    | Azure region where the dedicated instance is hosted          | `eastus`                                                                     | Unstructured |
| Customer Azure Subscription ID(s) | Subscription ID of each Azure subscription that needs access | `aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee`                                       | Customer     |

### Step 3a: Provide your Azure Subscription IDs

Provide Unstructured with the Azure Subscription IDs that need access to your dedicated instance. Unstructured will add these subscriptions to the visibility settings on the Private Link Service so connection requests from those subscriptions can be accepted.

### Step 3b: Create a Private Endpoint

In each Azure subscription that requires access, create a Private Endpoint targeting Unstructured's Private Link Service. [Learn how](https://learn.microsoft.com/en-us/azure/private-link/create-private-endpoint-portal).

When creating the endpoint:

* Use the Private Link Service Resource ID provided by Unstructured in Step 2
* Deploy it in a VNet and subnet in the same Azure region as your dedicated instance
* Confirm the subnet's [private endpoint network policies](https://learn.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy) are configured to allow the Private Endpoint to deploy. New subnets default to allowing this; on existing subnets, verify that your network security group (NSG) and route table (UDR) policies do not block the Private Endpoint

Creating the Private Endpoint sends a connection request to Unstructured.

### Step 3c: Unstructured approves your connection request

No action is required from you during this step. Unstructured will approve your Private Endpoint connection request.

### Step 3d: Configure DNS

Configure DNS so your dedicated instance hostname resolves to the private IP of your Private Endpoint. The right approach depends on your authoritative DNS and where your clients connect from.

For setup details across the most common scenarios — clients in your Azure VNet, on-premises clients reaching the platform over ExpressRoute or VPN, and clients connecting through a Zero Trust Network — see [Configure DNS for Azure Private Link](/business/azure/azure-privatelink/dns-privatelink).

## Step 4: Connect to your data sources and destinations

Once your users can access the Unstructured Platform, connect the platform to your data sources and destinations. The approach depends on the type of service.

* **Azure-managed services** (Blob Storage, ADLS Gen2, Azure OpenAI, AI Search, Azure SQL, Cosmos DB, PostgreSQL, MySQL, Synapse, Azure Databricks): See [Connecting to Azure-managed services](/business/azure/azure-privatelink/connect-to-services)
* **Customer-managed services on Azure** (self-hosted applications fronted by an Azure Standard Load Balancer): See [Connecting to customer-managed services on Azure](/business/azure/azure-privatelink/connect-to-customer-managed-services)

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Enterprise Connect for Azure
Source: https://docs.unstructured.io/business/azure/enterprise-connect

Enterprise Connect lets Unstructured connectors access Azure resources by authenticating as a customer-configured App Registration using short-lived access tokens.

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to have it enabled on your instance.
</Note>

## Enterprise Connect credential model

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored.

A **federated identity credential** is a trust configuration in Microsoft Entra ID. An **App Registration** is an application identity in Microsoft Entra ID.

## Supported Azure connectors

Enterprise Connect supports the following Azure connectors:

* Azure AI Search
* Azure Blob Storage

## How Enterprise Connect works

Enterprise Connect requires setup in your Azure subscription. You create an App Registration and add a federated identity credential configured to trust tokens issued by Unstructured. You then assign a role-based access control (RBAC) role on each Azure resource you want the connector to access. No secrets or passwords are exchanged during setup.

When a workflow runs, Unstructured exchanges that short-lived token with Microsoft Entra ID (formerly Azure AD) for an access token. Unstructured uses the token for that operation only and never stores it. Tokens expire automatically.

## Setup requirements

Enterprise Connect requires the following:

* A dedicated instance provisioned for your Unstructured account, with Enterprise Connect enabled. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
* An Azure subscription with permissions to create App Registrations and add Azure RBAC role assignments.

## Configure your connector

Select your preferred interface (Pipelines or API) to view setup instructions for your connector.

<Note>
  A connector configured with Enterprise Connect is available only in the workspace where it was created. It is not accessible from other workspaces in your Unstructured account.
</Note>

| Connector          | Source                                                                                                                                                                                            | Destination                                                                                                                                                                                                 |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Azure AI Search    | N/A                                                                                                                                                                                               | [Pipelines](/pipelines/destinations/azure-ai-search#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/destinations/azure-ai-search#set-up-enterprise-connect-authentication)       |
| Azure Blob Storage | [Pipelines](/pipelines/sources/azure-blob-storage#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/sources/azure-blob-storage#set-up-enterprise-connect-authentication) | [Pipelines](/pipelines/destinations/azure-blob-storage#set-up-enterprise-connect-authentication) \| [API](/api-reference/workflow/destinations/azure-blob-storage#set-up-enterprise-connect-authentication) |


# Onboarding in-VPC on your Microsoft Azure platform
Source: https://docs.unstructured.io/business/azure/onboarding-in-vpc



<Note>
  The following information applies only to in-VPC deployments of [Unstructured Business](/business/overview).

  For dedicated instance deployments of Unstructured Business, contact your Unstructured sales representative,
  or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

  For dedicated instance deployments of Unstructured Business that also use PrivateLink, see
  the [onboarding process for Azure Private Link](/business/azure/azure-privatelink/onboarding-di-pl).
</Note>

After your organization has signed the **Business** account agreement with Unstructured, a member of the Unstructured technical enablement team will reach out to you to begin the
deployment onboarding process. To streamline this process, you are encouraged to begin setting up your target environment as soon as possible. Choose one of the following setup options:

* [Do it all for me](#do-it-all-for-me): Have Unstructured set up the required infrastructure in your AWS account and then deploy Unstructured Pipelines and the API into that newly created infrastructure.
* [Bring my own infrastructure](#bring-my-own-infrastructure): Set up the required infrastructure yourself in your AWS account, and then have Unstructured deploy Unstructured Pipelines and the API into your existing infrastructure.

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).

## Do it all for me

If you want Unstructured to set up the required infrastructure for you into your Azure account and then deploy Unstructured Pipelines and the API into that newly created infrastrucrure, then provide your Unstructured sales representative or technical enablement contact with
the access credentials for a Microsoft Entra ID user or service principal in your Azure account that has the following required permissions.

### Subscription and resource group

* `Microsoft.Resources/subscriptions/resourceGroups/write` (to create the resource group)
* `Microsoft.Resources/subscriptions/resourceGroups/read` (to read the resource group)

### VNet and networking

* `Microsoft.Network/virtualNetworks/write` (to create the VNet)
* `Microsoft.Network/virtualNetworks/read` (to read the VNet)
* `Microsoft.Network/publicIPAddresses/write` (to create the public IPs)
* `Microsoft.Network/publicIPAddresses/read` (to read the public IPs)
* `Microsoft.Network/natGateways/write` (to create the NAT Gateway)
* `Microsoft.Network/natGateways/read` (to read the NAT Gateway)
* `Microsoft.Network/routeTables/write` (to create the route tables)
* `Microsoft.Network/routeTables/read` (to read the route tables)
* `Microsoft.Network/networkSecurityGroups/write` (to create the Network Security Groups)
* `Microsoft.Network/networkSecurityGroups/read` (to read the Network Security Groups)

### AKS cluster

* `Microsoft.ContainerService/managedClusters/write` (to create the AKS cluster)
* `Microsoft.ContainerService/managedClusters/read` (to read the AKS cluster)
* `Microsoft.ContainerService/agentPools/write` (to create the node pools)
* `Microsoft.ContainerService/agentPools/read` (to read the node pools)

### Managed identities and RBAC

* `Microsoft.ManagedIdentity/userAssignedIdentities/write` (to create the managed identities)
* `Microsoft.ManagedIdentity/userAssignedIdentities/read` (to read managed identities)
* Assign built-in roles such as:

  * **Contributor** or scoped **Network Contributor** for the AKS cluster identity
  * **Monitoring Metrics Publisher**, **AcrPull**, and **Storage Blob Data Reader** for the node pool identity
  * **Storage Blob Data Contributor** for workload identities

### Kubernetes add-ons

Permissions depend on the Helm/YAML installation, but Azure RBAC integration requires `Microsoft.ContainerService/managedClusters/accessProfiles/*/read` (to access kubeconfig)

### Storage class

* `Microsoft.Storage/storageAccounts/write` (to create the storage account for CSI driver provisioning)
* `Microsoft.Storage/storageAccounts/read`

### PostgreSQL database

* `Microsoft.DBforPostgreSQL/flexibleServers/write` (to create the PostgreSQL server)
* `Microsoft.DBforPostgreSQL/flexibleServers/read`
* NSG permissions for database access: allow traffic from the VNet CIDR

## Bring my own infrastructure

If you want to set up the required infrastructure yourself, set things up as follows within your Azure account for Unstructured to deploy Unstructured Pipelines and the API into.

You must also provide your Unstructured sales representative or technical enablement contact with
the access credentials for an IAM user or service principal in your AWS account that has access to the target Azure Kubernetes Service (AKS) cluster to deploy the
Unstructured Pipelines and the API into.

### **Azure subscription and resource group**

* **Subscription**

  * Ensure you have access to a valid Azure subscription
  * You will need the `subscription_id` if deploying via CLI or Pulumi

* **Resource Group**

  * Name: `u10d-{env}-rg`
  * Region: e.g., `eastus2`
  * All resources (VNet, AKS, PostgreSQL, Storage, etc.) will be created inside this group

### **VNet and networking**

* **Virtual Network (VNet)**

  * Address space: `10.0.0.0/16`
  * DNS Hostnames: Enabled
  * DNS Support: Enabled

* **Internet Access**

  * Handled via Azure's default gateway and public IPs

* **Public Subnet**

  * Address: `10.0.0.0/24`
  * Assign Public IP: true
  * Availability Zone: `${region}a`

* **NAT Gateway + Public IP**

  * NAT Gateway in the public subnet
  * Public IP resource attached

* **Private Subnets (x2)**

  * Addresses: `10.0.1.0/24`, `10.0.2.0/24`
  * AZs: `${region}a` and `${region}b`

* **Route Tables**

  * Public: route `0.0.0.0/0` via internet
  * Private: route `0.0.0.0/0` via NAT Gateway

### **Managed identities and RBAC**

* **AKS Cluster Managed Identity**

  * Assign roles:

    * `Contributor` or more scoped role
    * `Network Contributor`

* **Node Pool Managed Identity**

  * Assign roles:

    * `Monitoring Metrics Publisher`
    * `AcrPull` (if using ACR)
    * `Storage Blob Data Reader`

* **Workload Identity Bindings (x3)**

  * Namespaces: `recommender`, `etl-operator`, `data-broker`
  * Use Azure AD Workload Identity Federation
  * Assign `Storage Blob Data Contributor` to required containers

### **AKS Cluster**

* **Control Plane**

  * Version: `1.31` or higher
  * API authorized IPs: optional
  * Private cluster networking recommended

* **Node Pool**

  * VM Size: `Standard_D16s_v5`
  * Disk Size: 100 GB
  * Desired Size: 2 (min: 2, max: 5)
  * SSH: Enabled via key pair
  * SSH key exported in PEM format

* **Network Security Groups**

  * Allow intra-cluster traffic (`10.0.0.0/16`)
  * Allow all egress

#### **Kubernetes Add-ons**

Install via Helm or YAML:

* **Workload Identity Webhook**
* **Metrics Server** — `v0.7.2`
* **Azure Disk CSI Driver**

  * Provisioner: `disk.csi.azure.com`

### **Storage class**

```yaml theme={null}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-disk-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
  kind: Managed
volumeBindingMode: WaitForFirstConsumer
```

### Secrets and ConfigMaps

After your infrastructure is set up, but before Unstructured can deploy Unstructured Pipelines and the API into your insfrastructure,
Unstructured will need to know the values of the following Secrets and ConfigMaps. These must be provided to Unstructured as a
set of YAML files in Kubernetes [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) and
[ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) format.

Capture these during setup

* DB host, username, password
* Container names
* SSH private key
* Auth secrets

The Secrets are as follows.

#### Blob storage credentials (Azure)

* `BLOB_STORAGE_ADAPTER_ACCOUNT_NAME`
* `BLOB_STORAGE_ADAPTER_ACCOUNT_KEY`
* `BLOB_STORAGE_ADAPTER_CONTAINER_REGION` (optional)

#### Database credentials

* `DB_USERNAME`
* `DB_PASSWORD`
* `DB_HOST`
* `DB_NAME`
* `DB_DATABASE`

#### Authentication

* `JWT_SECRET_KEY`
* `AUTH_STRATEGY`
* `SESSION_SECRET`
* `SHARED_SECRET`
* `KEYCLOAK_CLIENT_SECRET`
* `KEYCLOAK_ADMIN_SECRET`
* `KEYCLOAK_ADMIN`
* `KEYCLOAK_ADMIN_PASSWORD`
* `API_BEARER_TOKEN`

The ConfigMaps are as follows.

#### Blob storage settings

* `BLOB_STORAGE_ADAPTER_TYPE`: `azure`
* `BLOB_STORAGE_ADAPTER_BUCKET`
* `ETL_BLOB_CACHE_BUCKET_NAME`
* `ETL_API_BLOB_STORAGE_ADAPTER_BUCKET`
* `ETL_API_BLOB_STORAGE_ADAPTER_TYPE`: `azure`
* `ETL_API_DB_REMOTE_BUCKET_NAME`
* `ETL_API_JOB_STATUS_DEST_BUCKET_NAME`
* `JOB_STATUS_BUCKET_NAME`
* `JOB_DB_BUCKET_NAME`

#### Environment

* `ENV`, `ENVIRONMENT`
* `JOB_ENV`, `JOB_ENVIRONMENT`

#### Observability and OpenTelementry (OTel)

* `JOB_OTEL_EXPORTER_OTLP_ENDPOINT`
* `JOB_OTEL_METRICS_EXPORTER`
* `JOB_OTEL_TRACES_EXPORTER`
* `OTEL_EXPORTER_OTLP_ENDPOINT`
* `OTEL_METRICS_EXPORTER`
* `OTEL_TRACES_EXPORTER`

#### Unstructured API and authentication

* `UNSTRUCTURED_API_URL`
* `JWKS_URL`
* `JWT_ISSUER`
* `JWT_AUDIENCE`
* `SINGLE_PLANE_DEPLOYMENT`

#### Front end and dashboard

* `API_BASE_URL`
* `API_CLIENT_BASE_URL`
* `API_URL`
* `APM_SERVICE_NAME`
* `APM_SERVICE_NAME_CLIENT`
* `AUTH_STRATEGY`
* `FRONTEND_BASE_URL`
* `KEYCLOAK_CALLBACK_URL`
* `KEYCLOAK_CLIENT_ID`
* `KEYCLOAK_DOMAIN`
* `KEYCLOAK_REALM`
* `KEYCLOAK_SSL_ENABLED`
* `KEYCLOAK_TRUST_ISSUER`
* `PUBLIC_BASE_URL`
* `PUBLIC_RELEASE_CHANNEL`

#### Redis

* `REDIS_DSN`

#### Other

* `IMAGE_PULL_SECRETS`
* `PRIVATE_KEY_SECRETS_ADAPTER_TYPE`: `azure`
* `PRIVATE_KEY_SECRETS_ADAPTER_AZURE_REGION`
* `SECRETS_ADAPTER_TYPE`: `azure`
* `SECRETS_ADAPTER_AZURE_REGION`

The preceding Secrets and ConfigMaps must be added to the following files:

| File Name                             | Type      | Resource name              | Namespace      | Data keys                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------- | --------- | -------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data-broker-env-cm.yaml`             | ConfigMap | `data-broker-env`          | `api`          | `JOB_STATUS_BUCKET_NAME`, `JOB_DB_BUCKET_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `data-broker-env-secret.yaml`         | Secret    | `data-broker-env`          | `api`          | `BLOB_STORAGE_ADAPTER_ACCOUNT_NAME`, `BLOB_STORAGE_ADAPTER_ACCOUNT_KEY`, `BLOB_STORAGE_ADAPTER_CONTAINER_REGION`                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `dataplane-api-env-cm.yaml`           | Secret    | `dataplane-api-env`        | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `etl-operator-env-cm.yaml`            | ConfigMap | `etl-operator-env`         | `etl-operator` | `BLOB_STORAGE_ADAPTER_BUCKET`, `JOB_STATUS_BUCKET_NAME`, `JOB_DB_BUCKET_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`, `ENV`, `ENVIRONMENT`, `REDIS_DSN`, `ETL_API_BLOB_STORAGE_ADAPTER_BUCKET`, `ETL_API_BLOB_STORAGE_ADAPTER_TYPE`, `ETL_API_DB_REMOTE_BUCKET_NAME`, `ETL_API_JOB_STATUS_DEST_BUCKET_NAME` (x2), `ETL_BLOB_CACHE_BUCKET_NAME`, `IMAGE_PULL_SECRETS`, `JOB_ENV`, `JOB_ENVIRONMENT`, `JOB_OTEL_EXPORTER_OTLP_ENDPOINT`, `JOB_OTEL_METRICS_EXPORTER`, `JOB_OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_METRICS_EXPORTER`, `OTEL_TRACES_EXPORTER`, `UNSTRUCTURED_API_URL` |
| `etl-operator-env-secret.yaml`        | Secret    | `etl-operator-env`         | `etl-operator` | `BLOB_STORAGE_ADAPTER_ACCOUNT_NAME`, `BLOB_STORAGE_ADAPTER_ACCOUNT_KEY`, `BLOB_STORAGE_ADAPTER_CONTAINER_REGION`                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `frontend-env-cm.yaml`                | ConfigMap | `frontend-env`             | `www`          | `API_BASE_URL`, `API_CLIENT_BASE_URL`, `API_URL`, `APM_SERVICE_NAME`, `APM_SERVICE_NAME_CLIENT`, `AUTH_STRATEGY`, `ENV`, `FRONTEND_BASE_URL`, `KEYCLOAK_CALLBACK_URL`, `KEYCLOAK_CLIENT_ID`, `KEYCLOAK_DOMAIN`, `KEYCLOAK_REALM`, `KEYCLOAK_SSL_ENABLED`, `KEYCLOAK_TRUST_ISSUER`, `PUBLIC_BASE_URL`, `PUBLIC_RELEASE_CHANNEL`, `SENTRY_DSN`, `SENTRY_SAMPLE_RATE`, `WORKFLOW_NODE_EDITOR_FF_REQUEST_FORM`, `CUSTOM_WORKFLOW_FF_REQUEST_FORM`                                                                                                                                                |
| `frontend-env-secret.yaml`            | Secret    | `frontend-env`             | `www`          | `API_BEARER_TOKEN`, `KEYCLOAK_ADMIN_SECRET`, `KEYCLOAK_CLIENT_SECRET`, `SESSION_SECRET`, `SHARED_SECRET`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `keycloak-secret.yaml`                | Secret    | `phasetwo-keycloak-env`    | `www`          | `KEYCLOAK_ADMIN`, `KEYCLOAK_ADMIN_PASSWORD`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `platform-api-env-cm.yaml`            | ConfigMap | `platform-api-env`         | `api`          | `JWKS_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `SINGLE_PLANE_DEPLOYMENT`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `platform-api-env-secret.yaml`        | Secret    | `platform-api-env`         | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`, `DB_DATABASE`, `JWT_SECRET_KEY`, `AUTH_STRATEGY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `recommender-env-cm.yaml`             | ConfigMap | `recommender-env`          | `recommender`  | `BLOB_STORAGE_ADAPTER_TYPE`, `ETL_BLOB_CACHE_BUCKET_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `recommender-env-secret.yaml`         | Secret    | `recommender-env`          | `recommender`  | `BLOB_STORAGE_ADAPTER_ACCOUNT_NAME`, `BLOB_STORAGE_ADAPTER_ACCOUNT_KEY`, `BLOB_STORAGE_ADAPTER_CONTAINER_REGION`                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `secret-provider-api-env-cm.yaml`     | ConfigMap | `secrets-provider-api-env` | `secrets`      | `ENV`, `ENVIRONMENT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_METRICS_EXPORTER`, `OTEL_TRACES_EXPORTER`, `PRIVATE_KEY_SECRETS_ADAPTER_AZURE_REGION`, `PRIVATE_KEY_SECRETS_ADAPTER_TYPE`, `SECRETS_ADAPTER_AZURE_REGION`, `SECRETS_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                 |
| `secret-provider-api-env-secret.yaml` | Secret    | `secrets-provider-api-env` | `secrets`      | `BLOB_STORAGE_ADAPTER_ACCOUNT_NAME`, `BLOB_STORAGE_ADAPTER_ACCOUNT_KEY`, `BLOB_STORAGE_ADAPTER_CONTAINER_REGION`                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `usage-collector-env-secret.yaml`     | Secret    | `usage-collector-env`      | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

For example, for the `data-broker-env-cm.yaml` ConfigMap file, the contents would look like this:

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: data-broker-env
  namespace: api
data:
  JOB_STATUS_BUCKET_NAME: "<your-value>"
  JOB_DB_BUCKET_NAME: "<your-value>"
  BLOB_STORAGE_ADAPTER_TYPE: "<your-value>"
```

The `data-broker-env-secret.yaml` Secret file would look like this:

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: data-broker-env
  namespace: api
type: Opaque
stringData:
  BLOB_STORAGE_ADAPTER_ACCOUNT_NAME: "<your-value>"
  BLOB_STORAGE_ADAPTER_ACCOUNT_KEY: "<your-value>"
  BLOB_STORAGE_ADAPTER_CONTAINER_REGION: "<your-value>"
```


# Unstructured Business on Microsoft Azure (Azure)
Source: https://docs.unstructured.io/business/azure/overview



## Getting started

To get started with a **Business** account deployment, your organization must first sign a **Business** account agreement with Unstructured.
This agreement outlines the terms and conditions for your organization to use Unstructured.

To begin the **Business** account agreement process, contact your Unstructured sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

After your organization has signed the **Business** account agreement, a member of the Unstructured technical enablement team will reach out to you to begin the
deployment onboarding process.

While you wait, you can get started by following the onboarding checklist for your deployment type, if it applies to you:

* For in-VPC deployments, follow [onboarding in-VPC on Azure](/business/azure/onboarding-in-vpc).

* For dedicated instance deployments with Azure Private Link, follow [onboarding a dedicated instance](/business/azure/azure-privatelink/onboarding-di-pl).

## Questions? Need help?

Contact your Unstructured account team or open a ticket in the [Unstructured Support Portal](https://support.unstructured.io/).


# Concepts and architecture
Source: https://docs.unstructured.io/business/dedicated-instances/architecture



<Note>
  This topic uses *private connectivity* as a general term for AWS PrivateLink and Azure Private Link.
</Note>

A **dedicated instance** is an isolated Unstructured deployment. When private connectivity is enabled, traffic between the customer environment and Unstructured stays on private network paths instead of traversing the public internet.

The diagrams below show the architecture for dedicated instances with private connectivity enabled.

### AWS dedicated instance with AWS PrivateLink

<img alt="AWS dedicated instance with AWS PrivateLink architecture" />

### Azure dedicated instance with Azure Private Link

<img alt="Azure dedicated instance with Azure Private Link architecture" />

## Private connectivity by cloud provider

**AWS private connectivity** uses AWS PrivateLink interface endpoints to connect the customer VPC to the Unstructured private network without traversing the public internet.

**Azure private connectivity** uses Azure Private Link to connect the customer VNet to the Unstructured private network while keeping traffic on the Microsoft backbone instead of the public internet.

## Traffic directions

Private connectivity can be configured in one or both directions:

* **Customer → Unstructured** — Customer users and applications access Unstructured Pipelines and the API through private endpoints in the customer VPC or VNet.
* **Unstructured → Customer** — Unstructured accesses customer data sources, such as S3 buckets, databases, and vector stores, through private endpoints in the Unstructured VPC or VNet.

If you enforce strict outbound controls, configure both directions so traffic remains private end to end.


# Before you begin
Source: https://docs.unstructured.io/business/dedicated-instances/before-you-begin



Deploying a dedicated instance requires coordination across multiple teams.
Identifying the right people before onboarding starts will help avoid delays
in your deployment.

## Who to involve

### Dedicated instance administrator

This person will be the primary administrator of your Unstructured account.
They handle initial account setup, user management, and day-to-day platform
configuration.

You will need to provide their name and email address to Unstructured at the
start of onboarding.

***

### Cloud infrastructure engineer

This person performs the technical configuration required to establish private
connectivity between your environment and your dedicated instance. They need
hands-on access to your cloud account and the ability to make network and DNS
changes.

**AWS — they may need to be able to:**

* Create and manage VPC Endpoints and security groups
* Create and configure Route 53 Private Hosted Zones and DNS records

**Azure — they may need to be able to:**

* Create and manage Private Endpoints
* Create and manage Private DNS Zones

**Common titles:** Cloud Engineer, Infrastructure Engineer, Network Engineer,
SRE, AWS or Azure Administrator

***

### IAM or security engineer

If you plan to connect the platform to your data sources — for example, S3
buckets or Azure Blob Storage — someone will need to configure the roles and
policies that grant the platform the necessary access.

**AWS — they may need to be able to:**

* Create and manage IAM roles and policies
* Update S3 bucket policies if cross-account access is required

**Azure — they may need to be able to:**

* Assign RBAC roles to Unstructured's service principal on the relevant resources

**Common titles:** IAM Engineer, Cloud Security Engineer, AWS Administrator,
Security Engineer

***

### Security and compliance reviewer

Certain changes required to securely connect your environment to your dedicated instance may require security review or change management approval. Loop this person in early so approvals do not block progress.

See [What may require security review](#what-may-require-security-review) below.

**Common titles:** Information Security Engineer, Security Architect, Compliance
Officer, CISO or delegate

***

## What to prepare before onboarding

Before onboarding begins, Unstructured will ask you to provide the following
information to provision your dedicated instance.

| Information Required  | Description                                                                                                                                                                                            |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Name of administrator | Primary administrator of the account (others can be added later)                                                                                                                                       |
| Admin email address   | Email address of the primary administrator                                                                                                                                                             |
| Cloud provider        | AWS or Azure                                                                                                                                                                                           |
| Cloud region          | See [supported cloud providers and regions](/business/dedicated-instances/overview#supported-cloud-providers-and-regions)                                                                              |
| Private connectivity  | Whether you require private connectivity (PrivateLink on AWS, Private Link on Azure). This determines your dedicated instance URL format and what network configuration is required during onboarding. |
| Desired URL           | With private connectivity: `<companyname>.privatelink.unstructuredapp.io` — Without private connectivity: `<companyname>.unstructuredapp.io`                                                           |
| VLM provider          | Anthropic, Bedrock, OpenAI, Azure OpenAI, etc.                                                                                                                                                         |
| VLM models            | For example, Claude Sonnet 4.5, GPT-4o, or Gemini 2.0 Flash                                                                                                                                            |
| Embedding models      | For example, Titan or Granite. Ensure that you select the correct dimensions.                                                                                                                          |
| Desired connectors    | The data sources and destinations you plan to connect to the platform. This helps Unstructured plan connectivity requirements according to your prioritized use cases.                                 |

<Note>
  Dedicated instance deployments require you to supply your own AI provider credentials — Unstructured does not provide shared API keys. You can use any supported provider. Your choice determines which models are available, and which providers support private endpoint access if you need to keep model traffic off the public internet. For details, see [AI providers](/business/ai-providers).
</Note>

***

## What may require security review

The following items commonly require review or change management approval in
enterprise organizations. Identify sign-off owners before onboarding starts.

| Item                                            | Why it may require review                                                                                                                                                                                                                                                               |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Private endpoint or VPC Endpoint connectivity   | Establishing a private network connection to an external account or subscription is a network boundary change that may require network or security team approval. On AWS this involves creating VPC Endpoints; on Azure it involves creating or approving Private Endpoint connections. |
| DNS configuration changes                       | Adding private hosted zones or modifying DNS records often requires change management approval in regulated environments.                                                                                                                                                               |
| IAM or RBAC role changes for data source access | Granting any external system access to storage or data services typically requires security team review. On AWS this involves IAM policy and role changes; on Azure it involves RBAC role assignments to Unstructured's service principal.                                              |

For more information about the dedicated instance deployment model to share with your security team, see:

* [Architecture](/business/dedicated-instances/architecture)
* [Security model](/business/dedicated-instances/security/security-model)
* [Security FAQ](/business/dedicated-instances/security/security-faq)


# Dedicated instance overview
Source: https://docs.unstructured.io/business/dedicated-instances/overview



A dedicated instance is a private, isolated deployment hosted by Unstructured for organizations that require stronger network boundaries, data isolation, and compliance controls. We can securely connect to your data sources and destinations and process your data without traversing the public internet when dedicated instances are paired with private connectivity.

The following 4-minute video provides an overview of dedicated instances.

<iframe title="YouTube video player" />

Use this section to help your teams evaluate, secure, and implement a dedicated instance in the Unstructured platform.

This section covers:

* Supported cloud providers and regions.
* Dedicated instance architecture.
* Who to involve and what to prepare before onboarding begins.
* Requirements and known limitations.
* Security model, shared responsibility, and compliance considerations.

## Cloud service provider (CSP) support

Unstructured supports dedicated instances on Amazon Web Services (AWS) and Microsoft Azure (Azure). This section covers architecture and configuration information that generally applies to dedicated instances on both cloud providers.

For provider-specific onboarding guidance, see:

* AWS — [Onboarding your dedicated instance + AWS PrivateLink](/business/aws/aws-privatelink/onboarding-di-pl)
* Azure — [Onboarding your dedicated instance + Azure Private Link](/business/azure/azure-privatelink/onboarding-di-pl)

### Supported cloud providers and regions

| Cloud provider              | Availability            |
| --------------------------- | ----------------------- |
| Amazon Web Services (AWS)   | General availability    |
| Microsoft Azure             | General availability    |
| Google Cloud Platform (GCP) | Not currently supported |

#### Supported AWS regions

| Region name           | Region code      |
| --------------------- | ---------------- |
| US East (N. Virginia) | `us-east-1`      |
| US East (Ohio)        | `us-east-2`      |
| US West (Oregon)      | `us-west-2`      |
| Europe (Ireland)      | `eu-west-1`      |
| Europe (Stockholm)    | `eu-north-1`     |
| Europe (Frankfurt)    | `eu-central-1`   |
| Asia Pacific (Sydney) | `ap-southeast-2` |
| Asia Pacific (Mumbai) | `ap-south-1`     |
| Asia Pacific (Seoul)  | `ap-northeast-2` |

If your required AWS region is not listed, contact your Unstructured account representative.

#### Supported Azure regions

| Region name                 | Region code     |
| --------------------------- | --------------- |
| East US 2 (Virginia)        | `eastus2`       |
| West US 2 (Washington)      | `westus2`       |
| North Europe (Ireland)      | `northeurope`   |
| West Europe (Netherlands)   | `westeurope`    |
| Southeast Asia (Singapore)  | `southeastasia` |
| Japan East (Tokyo, Saitama) | `japaneast`     |

If your required Azure region is not listed, contact your Unstructured account representative.

<Note>
  A dedicated instance is a regional service. It runs entirely in a single cloud region. Your private connectivity endpoints must be in the same region as your dedicated instance.
</Note>

<Note>
  This section uses *private connectivity* as a general term for AWS PrivateLink and Azure Private Link.
</Note>

## Private connectivity for your dedicated instance

Both AWS and Azure let you privately connect your Virtual Private Cloud (VPC) or Virtual Network (VNet) to your dedicated instance on the Unstructured platform without traversing the public internet.

Network filtering lets you block inbound and outbound internet traffic entirely or restrict it to specific IP addresses or CIDR ranges.

**Benefits**

* **Eliminate public exposure** - Traffic between your VPC or VNet and Unstructured stays on cloud-provider private networking instead of the public internet.
* **Stronger network isolation for sensitive workloads** - Support internal security and compliance requirements where private network access is preferred or required.
* **Simpler enterprise network integration** - Align with existing private connectivity patterns, including VPC or VNet isolation and private connectivity to on-premises networks.
* **Control and visibility** - Use private endpoints and private DNS to control traffic flow, enforce network-level access policies, and improve visibility into traffic paths and access patterns.


# Requirements and limitations
Source: https://docs.unstructured.io/business/dedicated-instances/requirements



<Note>
  This topic uses *private connectivity* as a general term for AWS PrivateLink and Azure Private Link.
</Note>

Dedicated Instances rely on cloud-provider private networking features and require baseline infrastructure components for a secure and successful deployment.

For supported cloud providers and regions, see [Dedicated instance overview](/business/dedicated-instances/overview#supported-cloud-providers-and-regions).

## Prerequisites

The following resources and capabilities are required to provision a dedicated instance.

#### AWS requirements

* VPC with private subnets
* Network Load Balancer for connections into the customer environment
* IAM permissions to create VPC endpoints, security groups, and Route 53 records

#### Azure requirements

* Virtual network with subnets configured for private endpoints
* Private endpoint network policies disabled on target subnets
* Standard Load Balancer for connections into the customer environment

## DNS requirements

Private connectivity requires DNS configuration to route traffic through private endpoints:

* **AWS:** Route 53 Private Hosted Zone, and on-premises DNS forwarding if needed
* **Azure:** Azure Private DNS Zone linked to the customer VNet
* **Both:** CNAME records that point to the private endpoint

For dedicated instances with private connectivity, Unstructured provides the DNS name in the format `<company>.privatelink.unstructuredapp.io`. For dedicated instances without private connectivity, the DNS name is in the format `<company>.unstructuredapp.io`.

## Known limitations

| Limitation                        | Notes                                                                                    |
| --------------------------------- | ---------------------------------------------------------------------------------------- |
| Cross-Region private connectivity | Not supported. Your VPC Endpoints must be in the same region as your dedicated instance. |
| Cross-CSP private connectivity    | Not supported, for example AWS to Azure                                                  |
| VPC Peering                       | Not supported as an alternative to private connectivity                                  |


# Security FAQ
Source: https://docs.unstructured.io/business/dedicated-instances/security/security-faq



<Note>
  This topic uses *private connectivity* as a general term for AWS
  PrivateLink and Azure Private Link.
</Note>

**Is my dedicated instance single-tenant?**

Yes. Each dedicated instance is single-tenant and logically isolated.
It is not shared with other customers at the application or network
level.

**Does traffic ever traverse the public internet?**

With private connectivity enabled, no traffic traverses the public internet by
default. Inbound access is restricted to your private network, and outbound
connections to your data sources and destinations use PrivateLink. Public egress
is blocked unless explicitly enabled.

For use cases where private connectivity is not supported — such as connecting
to SaaS applications — public egress can be enabled, but only if the customer
explicitly requests it by opening a support ticket with the [Unstructured Support Portal](https://support.unstructured.io/). It is not enabled automatically. See connector support for
[AWS](/business/aws/aws-privatelink/connector-support) and
[Azure](/business/azure/azure-privatelink/connector-support) to understand which
connectors support private connectivity.

Without private connectivity, the platform is accessible over the public
internet via HTTPS. Public ingress can be restricted to specific IP addresses
or CIDR ranges on request.

**Is data encrypted?**

Yes. Data is encrypted in the following ways:

* **In transit:** All traffic uses TLS 1.2 or higher.
* **Internally:** Mutual TLS (mTLS) is used between platform services.
* **In cloud storage:** Encryption policies are enforced at the
  storage layer.
* **At rest:** Data is encrypted at rest using the cloud provider's native key management service.

**Can Unstructured access my data?**

Unstructured personnel do not access customer data as part of normal
operations. Access to customer data sources is explicitly configured,
limited to the minimum permissions required, and governed by
customer-controlled IAM, RBAC, and resource policies. Any exceptional
access, such as for troubleshooting, is controlled and audited.

**Is customer data used to train models?**

No. Customer data processed within a dedicated instance is not used to
train models.

**How is access to the platform controlled?**

* Customers manage user access to Unstructured Pipelines and the APIs.
* Network access can be restricted by using private endpoints,
  security groups, and firewall rules.
* DNS resolution can be kept private and scoped to customer networks.

**What does private connectivity protect against?**

Private connectivity helps protect against:

* Exposure to the public internet
* Unintended inbound network access
* DNS-based traffic interception

Private connectivity does not protect against:

* Application-layer vulnerabilities
* Misconfigured IAM or RBAC policies
* Compromised credentials

**How long is customer data retained?**

Unstructured does not store customer data long-term. Customer
artifacts are used ephemerally during workflow execution and removed
upon completion or failure. Transformed data is written to the
configured destination. The data that persists in the Unstructured
platform is limited to connector and workflow configuration metadata,
connector keys stored in a secret store, and workflow execution
telemetry, which does not contain file data.

**What compliance standards does Unstructured support?**

Unstructured maintains industry-standard security and compliance
programs, including HIPAA compliance architecture, ITAR readiness
with geo-blocking through AWS WAF, and SOC 2 controls. Detailed
compliance artifacts are available in the [trust
portal](https://trust.unstructured.io/).


# Security model
Source: https://docs.unstructured.io/business/dedicated-instances/security/security-model



<Note>
  This topic uses *private connectivity* as a general term for AWS PrivateLink and Azure Private Link.
</Note>

This page explains how dedicated instances protect data through private connectivity, encryption, and network access controls.

## Encryption

Dedicated Instances use encryption in transit across external and internal service boundaries. All supported connections use TLS 1.2 or higher.

| Layer                                        | Encryption                                                                                   |
| -------------------------------------------- | -------------------------------------------------------------------------------------------- |
| API endpoints                                | TLS 1.2 or higher with certificate validation                                                |
| Cloud storage connections (S3, Blob Storage) | TLS 1.2 or higher, with bucket or container policies used to enforce encryption requirements |
| Internal service mesh                        | mTLS between microservices                                                                   |

## Security without private connectivity (internet-facing mode)

Customers who deploy a dedicated instance without private connectivity access the Unstructured platform over the public internet via HTTPS.

The following security measures and connectivity options apply:

* **TLS 1.2+** for all traffic in transit.

* **AWS WAF** (Web Application Firewall) provides DDoS protection, rate limiting, and optional geo-blocking for ITAR compliance.

* **IP Allowlisting** - restricts Unstructured platform access to specific source IP addresses or CIDR ranges. Submit a support ticket with your IP list to enable this.

* **Independent paths** - allow you to use Pipelines and the API over the public internet while Unstructured uses private connectivity to reach your data sources. The two paths are configured separately.

## Security with private connectivity

When private connectivity is enabled, traffic between your environment and the Unstructured platform stays on cloud-provider private networking. Service endpoints resolve to private IP addresses, and inbound access from the public internet is blocked.

### What private connectivity does and does not protect

**Private connectivity helps protect:**

* Network traffic from public internet exposure.
* Data in transit between VPCs and VNets.
* DNS resolution of service endpoints.

**Private connectivity does not protect against:**

* Application-layer vulnerabilities.
* Misconfigured Identity and Access Management (IAM) or Role-Based Access Control (RBAC) policies.
* Compromised credentials.

## Network access controls

Network access controls determine whether traffic between the Unstructured platform and your cloud environment may traverse the public internet. Your cloud environment is the VPC or VNet that hosts your data sources and related resources. This section describes the default inbound and outbound traffic rules and how to request exceptions when a use case requires them.

**Default configuration:**

| Traffic Direction | Without private connectivity                                                                                                                                                               | With private connectivity                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Public Ingress    | **Enabled** — The platform is accessible over the public internet via HTTPS. Customers can request to restrict access to specific IP addresses or CIDR ranges by opening a support ticket. | **Blocked** — All inbound traffic from the public internet is denied. Access is only available via private connectivity.                                                                                                                                                                                                                                                                                                          |
| Public Egress     | **Enabled** — Outbound traffic to the public internet is permitted.                                                                                                                        | **Blocked by default** — Supported connectors connect via PrivateLink, keeping all traffic off the public internet. For use cases where private connectivity is not supported — such as SaaS connectors — public egress can be enabled on request by opening a support ticket. See connector support for [AWS](/business/aws/aws-privatelink/connector-support) and [Azure](/business/azure/azure-privatelink/connector-support). |


# Shared responsibility model
Source: https://docs.unstructured.io/business/dedicated-instances/security/shared-responsibility-model



A shared responsibility model explains how security and operational responsibilities are divided between Unstructured and the customer.  Unstructured is responsible for the infrastructure and software that it operates. The customer is responsible for its accounts, data, identities, access policies, and the network resources that it controls.

This table shows responsibilities for traffic from the customer environment to the Unstructured platform, which includes access to Pipelines and the API.

**Customer → Unstructured (Access to Unstructured Pipelines and the API)**

| Responsibility                             | Customer | Unstructured |
| ------------------------------------------ | -------- | ------------ |
| VPC / VNet Configuration                   | ✔        |              |
| VPC Endpoint / Private Endpoint creation   | ✔        |              |
| DNS Configuration                          | ✔        |              |
| Security Group / NSG Rules                 | ✔        |              |
| VPC Endpoint Service Creation              |          | ✔            |
| Endpoint Connection Approval               |          | ✔            |
| Load Balancer and Target Configuration     |          | ✔            |
| Platform Security & Certificate Management |          | ✔            |

This table shows responsibilities for traffic from Unstructured to customer-managed data sources in the customer cloud environment.

**Unstructured → Customer (Access to customer-managed data sources)**

| Responsibility                           | Customer | Unstructured |
| ---------------------------------------- | -------- | ------------ |
| VPC Endpoint / Private Endpoint Creation |          | ✔            |
| DNS Configuration                        |          | ✔            |
| Security Group / NSG Rules               | ✔        |              |
| VPC Endpoint Service Creation            | ✔        |              |
| Load Balancer and Target Configuration   | ✔        |              |
| Endpoint Connection Approval             | ✔        |              |
| Bucket / Container Policies              | ✔        |              |

<Note>
  Each party is responsible only for the cloud resources that it owns and operates.
</Note>


# Onboarding in-VPC on your Google Cloud Platform (GCP)
Source: https://docs.unstructured.io/business/gcp/onboarding-in-vpc



<Note>
  The following information applies only to in-VPC deployments of [Unstructured Business](/business/overview).

  For dedicated instance deployments of Unstructured Business, contact your Unstructured sales representative,
  or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

After your organization has signed the **Business** account agreement with Unstructured, a member of the Unstructured technical enablement team will reach out to you to begin the
deployment onboarding process. To streamline this process, you are encouraged to begin setting up your target environment as soon as possible. Choose one of the following setup options:

* [Do it all for me](#do-it-all-for-me): Have Unstructured set up the required infrastructure in your AWS account and then deploy Unstructured Pipelines and the API into that newly created infrastructure.
* [Bring my own infrastructure](#bring-my-own-infrastructure): Set up the required infrastructure yourself in your AWS account, and then have Unstructured deploy Unstructured Pipelines and the API into your existing infrastructure.

## Questions? Need help?

If you have questions or need help as you go, contact your Unstructured sales representative or technical enablement contact. If you do not know who they are,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io), and a member of the Unstructured sales or technical enablement teams
will get back to you as soon as possible.

## Do it all for me

If you want Unstructured to set up the required infrastructure for you in your GCP account and then deploy Unstructured Pipelines and the API into that newly created infrastructure, then provide your Unstructured sales representative or technical enablement contact with
the access credentials for an IAM user or service account in your GCP account that has the following required permissions:

### Core networking permissions

VPC/subnet management:

* `compute.networks.create`
* `compute.subnetworks.create`
* `compute.routers.create` (for Cloud NAT)
* `compute.addresses.create` (for NAT IPs)
* `compute.firewalls.create` (for intra-cluster traffic rules)

Shared VPC (if used):

* `compute.organizations.admin` (for the host project)
* `compute.networks.use` (for the service project)

### GKE cluster permissions

Control plane:

* `container.clusters.create`
* `container.clusters.update` (for private cluster settings)
* `compute.networks.useExternalIp` (for public endpoint access)

Node pools:

* `compute.instances.create`
* `compute.disks.create` (for node disks)
* `compute.instanceGroups.create` (for autoscaling)

IAM roles:

* For the GKE cluster SA service account: `roles/container.hostServiceAgentUser`
* For the node SA	service account: `roles/container.nodeServiceAccount`
* For the workload identity	service account: `roles/iam.workloadIdentityUser`

### Storage and database

GCS buckets:

* `storage.buckets.create`
* `storage.objects.create` (for versioning)
* `storage.buckets.update` (for encryption/lifecycle rules)

Cloud SQL:

* `cloudsql.instances.create`
* `cloudsql.instances.connect` (for private IPs)
* `vpcaccess.connectors.use` (if using Serverless VPC Access)

Persistent disks (CSI):

* `compute.disks.create` (for `pd.csi.storage.gke.io`)
* `compute.subnetworks.use` (for regional disks)

### Advanced configurations

Workload identity:

* `iam.serviceAccounts.getAccessToken` (for federated access)
* `iam.serviceAccounts.setIamPolicy` (to bind Kubernetes SAs to GCP SAs)

Cloud NAT:

* `compute.routers.update` (for NAT configuration)
* `compute.addresses.use` (for NAT IP allocation)

OS login/SSH:

* `compute.projects.setCommonInstanceMetadata` (for SSH key upload)
* `compute.instances.osAdminLogin`

### Minimum required roles

Project level:

* `roles/editor` (broad access, or scope with custom roles)

Scoped roles:

* `roles/compute.networkAdmin` (for VPC and subnets)
* `roles/container.admin` (for GKE)
* `roles/storage.admin` (for GCS)
* `roles/cloudsql.admin` (for Postgres)

## Bring my own infrastructure

If you want to set up the required infrastructure yourself, set things up as follows within your GCP account for Unstructured to deploy Unstructured Pipelines and the API into.

You must also provide your Unstructured sales representative or technical enablement contact with
the access credentials for an IAM user or service account in your GCP account that has access to the target Google Kubernetes Engine (GKE) cluster to deploy the
Unstructured Pipelines and the API into.

### **VPC and networking (GCP equivalent)**

* **VPC Network**

  * Name: `u10d-platform`
  * Subnet Mode: *Custom*
  * CIDR: `10.0.0.0/16`
  * DNS: Internal DNS supported by default

* **Internet Gateway**

  * GCP provides implicit internet access via default internet gateway

    (No need to explicitly create)

* **Public Subnet**

  * Subnet: `public-subnet` — `10.0.0.0/24`
  * Region: `${region}`
  * Enable external IPs on VM instances for internet access

* **NAT Gateway**

  * Use **Cloud NAT** attached to a **Cloud Router** in public subnet
  * Needed to provide egress internet access to private subnet instances

* **Private Subnets (x2)**

  * `private-subnet-a`: `10.0.1.0/24`, region `${region}-a`
  * `private-subnet-b`: `10.0.2.0/24`, region `${region}-b`

* **Routes**

  * Public subnet: default route `0.0.0.0/0` to Internet Gateway (via external IPs)
  * Private subnets: route `0.0.0.0/0` via Cloud NAT

### **IAM roles and policies**

* **GKE Cluster IAM Service Account**

  * Grant roles:

    * `roles/container.clusterAdmin`
    * `roles/compute.networkAdmin`

* **GKE Node IAM Service Account**

  * Grant roles:

    * `roles/container.nodeServiceAccount`
    * `roles/compute.viewer`
    * `roles/storage.objectViewer`

* **Workload Identity IAM Bindings (x3)**

  * Namespaces: `recommender`, `etl-operator`, `data-broker`
  * Use **Workload Identity Federation**
  * Bind GCP IAM Service Accounts to Kubernetes service accounts
  * Grant `roles/storage.objectAdmin` for access to GCS buckets

### **GKE cluster**

* **Control Plane**

  * Version: `1.31` or higher
  * Private Cluster: *Enabled*
  * Master Authorized Networks: your IP(s)
  * Enable Public Endpoint Access: Yes

* **Node Pool**

  * Machine type: `n2-standard-16`
  * Disk: 100GB, SSD (default boot disk)
  * Node count: min 2, max 5, autoscaling enabled
  * SSH access: via OS Login + SSH keys
  * SSH key: Add public key to instance metadata

* **Firewall Rules**

  * Allow:

    * Internal: `10.0.0.0/16`
    * Egress: all

  * Kubernetes master access to nodes

### **Kubernetes add-ons (installed via `kubectl` or Helm)**

* **Workload Identity Config**

* **Metrics Server**

  * Deployed manually (same version: `v0.7.2`)

* **GCP CSI Driver**

  * Provisioner: `pd.csi.storage.gke.io`
  * Role binding needed for controller SA

### **Storage class**

```yaml theme={null}
apiVersion: [storage.k8s.io/v1](http://storage.k8s.io/v1)
kind: StorageClass
metadata:
 name: pd-ssd
 annotations:
  [storageclass.kubernetes.io/is-default-class:](http://storageclass.kubernetes.io/is-default-class:) "true"
provisioner: [pd.csi.storage.gke.io](http://pd.csi.storage.gke.io/)
parameters:
 type: pd-ssd
 encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
```

### **Cloud SQL (Postgres)**

* **Private IP-enabled Cloud SQL instance**

  * Engine: Postgres 16
  * Size: `db-f1-micro` (or `db-custom-1-3840`)
  * Storage: 20GB
  * Credentials: Username/password
  * Private network: Use the private VPC

* **Cloud SQL Auth Proxy** or private VPC peering to connect from GKE

### **GCS Buckets**

* Buckets:

  * `u10d-{stack_name}-etl-blob-cache`
  * `u10d-{stack_name}-etl-job-db`
  * `u10d-{stack_name}-etl-job-status`
  * `u10d-{stack_name}-job-files`

* Config:

  * Versioning: Enabled
  * Encryption: Default (Google-managed key or CMEK if needed)
  * Lifecycle rule: Auto-delete / force destroy if needed (optional)

### **Keys**

* **SSH Key Pair**

  * Generate manually (`ssh-keygen -t rsa -b 4096`)
  * Upload public key to project metadata or OS Login
  * Export private key as PEM for automation

### Secrets and ConfigMaps

After your infrastructure is set up, but before Unstructured can deploy Unstructured Pipelines and the API into your insfrastructure,
Unstructured will need to know the values of the following Secrets and configuration mappings (also known as *ConfigMaps*).

The Secrets are as follows.

#### **Blob storage credentials**

* `BLOB_STORAGE_ADAPTER_GCP_SERVICE_ACCOUNT_KEY_JSON`
* `BLOB_STORAGE_ADAPTER_REGION_NAME`

#### **Database credentials**

* `DB_USERNAME`
* `DB_PASSWORD`
* `DB_HOST`
* `DB_NAME`
* `DB_DATABASE` (used in `platform-api` only)

#### **Authentication**

* `JWT_SECRET_KEY`
* `AUTH_STRATEGY` (sometimes encoded, sometimes not)
* `SESSION_SECRET`
* `SHARED_SECRET`
* `KEYCLOAK_CLIENT_SECRET`
* `KEYCLOAK_ADMIN_SECRET`
* `KEYCLOAK_ADMIN`
* `KEYCLOAK_ADMIN_PASSWORD`
* `API_BEARER_TOKEN`

The ConfigMaps are as follows.

#### **Blob storage settings**

* `BLOB_STORAGE_ADAPTER_TYPE` (always `gcp`  for GCP)
* `BLOB_STORAGE_ADAPTER_BUCKET`
* `ETL_BLOB_CACHE_BUCKET_NAME`
* `ETL_API_BLOB_STORAGE_ADAPTER_BUCKET`
* `ETL_API_BLOB_STORAGE_ADAPTER_TYPE`
* `ETL_API_DB_REMOTE_BUCKET_NAME`
* `ETL_API_JOB_STATUS_DEST_BUCKET_NAME`
* `JOB_STATUS_BUCKET_NAME`
* `JOB_DB_BUCKET_NAME`

#### **Environment**

* `ENV`
* `ENVIRONMENT`
* `JOB_ENV`
* `JOB_ENVIRONMENT`

#### **Observability and OpenTelemetry (OTel)**

* `JOB_OTEL_EXPORTER_OTLP_ENDPOINT`
* `JOB_OTEL_METRICS_EXPORTER`
* `JOB_OTEL_TRACES_EXPORTER`
* `OTEL_EXPORTER_OTLP_ENDPOINT`
* `OTEL_METRICS_EXPORTER`
* `OTEL_TRACES_EXPORTER`

#### **Unstructured API and authentication**

* `UNSTRUCTURED_API_URL`
* `JWKS_URL`
* `JWT_ISSUER`
* `JWT_AUDIENCE`
* `SINGLE_PLANE_DEPLOYMENT`

#### **Front end and dashboard**

* `API_BASE_URL`
* `API_CLIENT_BASE_URL`
* `API_URL`
* `APM_SERVICE_NAME`
* `APM_SERVICE_NAME_CLIENT`
* `AUTH_STRATEGY`
* `FRONTEND_BASE_URL`
* `KEYCLOAK_CALLBACK_URL`
* `KEYCLOAK_CLIENT_ID`
* `KEYCLOAK_DOMAIN`
* `KEYCLOAK_REALM`
* `KEYCLOAK_SSL_ENABLED`
* `KEYCLOAK_TRUST_ISSUER`
* `PUBLIC_BASE_URL`
* `PUBLIC_RELEASE_CHANNEL`

#### **Sentry & Feature Flags**

* `SENTRY_DSN`
* `SENTRY_SAMPLE_RATE`
* `WORKFLOW_NODE_EDITOR_FF_REQUEST_FORM`
* `CUSTOM_WORKFLOW_FF_REQUEST_FORM`

#### **Redis**

* `REDIS_DSN`

#### **Other**

* `IMAGE_PULL_SECRETS`
* `PRIVATE_KEY_SECRETS_ADAPTER_TYPE`
* `PRIVATE_KEY_SECRETS_ADAPTER_GCP_REGION`
* `SECRETS_ADAPTER_TYPE`
* `SECRETS_ADAPTER_GCP_REGION`

The preceding Secrets and ConfigMaps must be added to the following files:

| File name                             | Type      | Resource name              | Namespace      | Data keys                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------- | --------- | -------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data-broker-env-cm.yaml`             | ConfigMap | `data-broker-env`          | `api`          | `JOB_STATUS_BUCKET_NAME`, `JOB_DB_BUCKET_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `data-broker-env-secret.yaml`         | Secret    | `data-broker-env`          | `api`          | `BLOB_STORAGE_ADAPTER_GCP_SERVICE_ACCOUNT_KEY_JSON`, `BLOB_STORAGE_ADAPTER_REGION_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `dataplane-api-env-cm.yaml`           | Secret    | `dataplane-api-env`        | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `etl-operator-env-cm.yaml`            | ConfigMap | `etl-operator-env`         | `etl-operator` | `BLOB_STORAGE_ADAPTER_BUCKET`, `JOB_STATUS_BUCKET_NAME`, `JOB_DB_BUCKET_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`, `ENV`, `ENVIRONMENT`, `REDIS_DSN`, `ETL_API_BLOB_STORAGE_ADAPTER_BUCKET`, `ETL_API_BLOB_STORAGE_ADAPTER_TYPE`, `ETL_API_DB_REMOTE_BUCKET_NAME`, `ETL_API_JOB_STATUS_DEST_BUCKET_NAME` (x2), `ETL_BLOB_CACHE_BUCKET_NAME`, `IMAGE_PULL_SECRETS`, `JOB_ENV`, `JOB_ENVIRONMENT`, `JOB_OTEL_EXPORTER_OTLP_ENDPOINT`, `JOB_OTEL_METRICS_EXPORTER`, `JOB_OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_METRICS_EXPORTER`, `OTEL_TRACES_EXPORTER`, `UNSTRUCTURED_API_URL` |
| `etl-operator-env-secret.yaml`        | Secret    | `etl-operator-env`         | `etl-operator` | `BLOB_STORAGE_ADAPTER_GCP_SERVICE_ACCOUNT_KEY_JSON`, `BLOB_STORAGE_ADAPTER_REGION_NAME,`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `frontend-env-cm.yaml`                | ConfigMap | `frontend-env`             | `www`          | `API_BASE_URL`, `API_CLIENT_BASE_URL`, `API_URL`, `APM_SERVICE_NAME`, `APM_SERVICE_NAME_CLIENT`, `AUTH_STRATEGY`, `ENV`, `FRONTEND_BASE_URL`, `KEYCLOAK_CALLBACK_URL`, `KEYCLOAK_CLIENT_ID`, `KEYCLOAK_DOMAIN`, `KEYCLOAK_REALM`, `KEYCLOAK_SSL_ENABLED`, `KEYCLOAK_TRUST_ISSUER`, `PUBLIC_BASE_URL`, `PUBLIC_RELEASE_CHANNEL`, `SENTRY_DSN`, `SENTRY_SAMPLE_RATE`, `WORKFLOW_NODE_EDITOR_FF_REQUEST_FORM`, `CUSTOM_WORKFLOW_FF_REQUEST_FORM`                                                                                                                                                |
| `frontend-env-secret.yaml`            | Secret    | `frontend-env`             | `www`          | `API_BEARER_TOKEN`, `KEYCLOAK_ADMIN_SECRET`, `KEYCLOAK_CLIENT_SECRET`, `SESSION_SECRET`, `SHARED_SECRET`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `keycloak-secret.yaml`                | Secret    | `phasetwo-keycloak-env`    | `www`          | `KEYCLOAK_ADMIN`, `KEYCLOAK_ADMIN_PASSWORD`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `platform-api-env-cm.yaml`            | ConfigMap | `platform-api-env`         | `api`          | `JWKS_URL`, `JWT_ISSUER`, `JWT_AUDIENCE`, `SINGLE_PLANE_DEPLOYMENT`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `platform-api-env-secret.yaml`        | Secret    | `platform-api-env`         | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`, `DB_DATABASE`, `JWT_SECRET_KEY`, `AUTH_STRATEGY`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `recommender-env-cm.yaml`             | ConfigMap | `recommender-env`          | `recommender`  | `BLOB_STORAGE_ADAPTER_TYPE`, `ETL_BLOB_CACHE_BUCKET_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `recommender-env-secret.yaml`         | Secret    | `recommender-env`          | `recommender`  | `BLOB_STORAGE_ADAPTER_GCP_SERVICE_ACCOUNT_KEY_JSON`, `BLOB_STORAGE_ADAPTER_REGION_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `secret-provider-api-env-cm.yaml`     | ConfigMap | `secrets-provider-api-env` | `secrets`      | `ENV`, `ENVIRONMENT`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_METRICS_EXPORTER`, `OTEL_TRACES_EXPORTER`, `PRIVATE_KEY_SECRETS_ADAPTER_GCP_REGION`, `PRIVATE_KEY_SECRETS_ADAPTER_TYPE`, `SECRETS_ADAPTER_GCP_REGION`, `SECRETS_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                     |
| `secret-provider-api-env-secret.yaml` | Secret    | `secrets-provider-api-env` | `secrets`      | `BLOB_STORAGE_ADAPTER_GCP_SERVICE_ACCOUNT_KEY_JSON`, `BLOB_STORAGE_ADAPTER_REGION_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `usage-collector-env-secret.yaml`     | Secret    | `usage-collector-env`      | `api`          | `DB_PASSWORD`, `DB_USERNAME`, `DB_HOST`, `DB_NAME`, `BLOB_STORAGE_ADAPTER_TYPE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

For example, for the `data-broker-env-cm.yaml` [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) file, the contents would look like this:

```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: data-broker-env
  namespace: api
data:
  JOB_STATUS_BUCKET_NAME: "<your-value>"
  JOB_DB_BUCKET_NAME: "<your-value>"
  BLOB_STORAGE_ADAPTER_TYPE: "<your-value>"
```

For the `data-broker-env-secret.yaml` [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) file, the contents would look like this:

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: data-broker-env
  namespace: api
type: Opaque
stringData:
  BLOB_STORAGE_ADAPTER_GCP_SERVICE_ACCOUNT_KEY_JSON: "<your-value>"
  BLOB_STORAGE_ADAPTER_REGION_NAME: "<your-value>"
```


# Unstructured Business in-VPC on Google Cloud Platform (GCP) - overview
Source: https://docs.unstructured.io/business/gcp/overview



<Note>
  The following information applies only to in-VPC deployments of [Unstructured Business](/business/overview).

  For dedicated instance deployments of Unstructured Business, contact your Unstructured sales representative,
  or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

## Getting started

To get started with a **Business** account deployment, your organization must first sign a **Business** account agreement with Unstructured.
This agreement outlines the terms and conditions for your organization to use Unstructured in a **Business** account environment.

To begin the **Business** account agreement process, contact your Unstructured sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

After your organization has signed the **Business** account agreement, a member of the Unstructured technical enablement team will reach out to you to begin the
deployment onboarding process. To streamline this process, you are encouraged to begin setting up your target environment as soon as possible. To do this,
see the [onboarding checkist](/business/gcp/onboarding-in-vpc).

## Questions? Need help?

Contact your Unstructured sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io), and a member of the Unstructured sales or support teams
will get back to you as soon as possible.


# Configure an identity provider
Source: https://docs.unstructured.io/business/idp/configuration



<Note>
  The following information applies only to dedicated instance and in-VPC deployments of [Unstructured Business](/business/overview).

  IdP integration is not available for Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** accounts.
</Note>

You can connect your organization's identity provider (IdP) to your Unstructured deployment through Unstructured Pipelines. After you connect your IdP, a single sign-on (SSO) option appears on your deployment's login page, and your IdP group memberships are available for access control.

## Prerequisites

You need site admin access to your Unstructured deployment and admin access to your organization's IdP. If you don't have site admin access, contact your Unstructured account team to have it provisioned. To learn how to designate site administrators through your IdP, see [Site administrator access](/business/idp/group-mappings#site-administrator-access).

The setup is a two-step exchange between Unstructured and your IdP. First, Unstructured generates a redirect URL when you enter an alias. You take that URL to your IdP and use it to register a new application. The application registration gives you the credentials you bring back to finish the connection in Unstructured. No credentials are shared with Unstructured outside of the Pipelines flow.

**For OIDC**, you will collect the following from your IdP application:

* Client ID
* Client secret
* Authorization URL
* Token URL
* JWKS URL

**For SAML**, you will collect the following from your IdP application:

* SSO URL
* Signing certificate

## Add an identity provider

1. If you are not already signed in, sign in to your Unstructured account as a site admin.

2. In the bottom-left corner, click your profile icon.

3. Click **Site Settings**.

4. On the **Identity Provider Configuration** page, click **New Identity Provider +**.

5. In the **Alias** field, enter a short, lowercase name to identify this IdP (for example, `okta-prod`).

6. A redirect URL is generated based on your alias. Click the copy icon to copy this URL.

   <Note>
     Paste this URL into your IdP provider's redirect or callback URI configuration before continuing.
   </Note>

7. Click **Continue**.

8. Under **Configuration**, select **OIDC** or **SAML** to match your IdP application's protocol.

9. If you selected OIDC, enter the following:
   * **Authorization URL**: The authorization endpoint from your IdP.
   * **Token URL**: The token endpoint from your IdP.
   * **Client ID**: The client ID from your IdP application.
   * **Client Secret**: The client secret from your IdP application.
   * **JWKS URL**: The JWKS endpoint from your IdP.

10. If you selected SAML, enter the following:
    * **SSO URL**: The single sign-on URL from your IdP.
    * **Signing Certificate**: Paste the signing certificate from your IdP.

11. Optionally, expand **Advanced settings** to configure additional options.

12. Click **Save Identity Provider**.

<Warning>
  For OIDC configurations, the client secret is hidden after you save. You cannot view it again.
</Warning>

After you save, a new SSO login option appears on your deployment's login page.

## Update an identity provider

1. If you are not already signed in, sign in to your Unstructured account as a site admin.
2. In the bottom-left corner, click your profile icon.
3. Click **Site Settings**.
4. On the **Identity Provider Configuration** page, click the IdP you want to update.
5. Update the fields as needed.
6. Click **Save Identity Provider**.

## Next steps

* To control which users can access your Unstructured account and what roles they receive, see [Group mappings](/business/idp/group-mappings).
* To learn about the roles available in Unstructured, see [Role-based access control (RBAC)](/pipelines/account/roles).


# IdP group mappings
Source: https://docs.unstructured.io/business/idp/group-mappings



<Note>
  The following information applies only to dedicated instance and in-VPC deployments of [Unstructured Business](/business/overview).

  IdP integration is not available for Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** accounts.
</Note>

If your organization's IdP is not yet connected, see [Configure an identity provider](/business/idp/configuration).

## Site administrator access

Site administrator is a deployment-wide role, separate from the organizational account and workspace roles covered on this page. It is **not** an option in the group mapping steps below: you cannot assign the site administrator role to an IdP group through Unstructured Pipelines.

To grant site administrator access through your identity provider:

1. Create a dedicated group in your IdP for your site administrators.
2. Add the users who should have site administrator access to that group.
3. Provide the group name to your Unstructured account team.

Once configured, any user in that group receives site administrator access when they sign in. Removing a user from the group revokes their access at their next sign-in.

To learn what the site administrator role can do, see [Role-based access control](/pipelines/account/roles#site-administrator-role).

## Add an IdP group to an organizational account

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that you want to add the IdP group to.

3. In the sidebar, above your user icon, click the **Settings** (gear) icon, and then click **Manage Account**.

4. In the top navigation bar, click **IdP Groups**.

5. Click **+ New Group**.

6. On the **Connect Group** page, for **Identity Provider Group**, type the name of the IdP group that you want to add, and then click **Continue**.

   <Note>
     You must type the name of the IdP group exactly as it appears in your IdP. Otherwise, Unstructured will not be able to
     successfully complete the connection to that IdP group through your IdP.
   </Note>

7. On the **Assign Account Role** page, for **Account Role**, select the name of the [organizational account role](/pipelines/account/roles#organizational-account-roles) that you want to assign to the IdP group for this
   organizational account, and then click **Continue**.

8. On the **Assign Workspaces** page, for **Workspaces and permissions**, select each workspace—and the
   [workspace role](/pipelines/account/roles#workspace-roles) for that workspace—that you want to assign to the IdP group, and then click **Add**.

   <Note>
     The **Assign Workspaces** page does not apply if the IdP group is assigned the **Super Administrator** organizational account role.
     This is because an IdP group with the **Super Administrator** role automatically has full access to all of the organizational account's workspaces.
   </Note>

9. Click **Save Group**.

The organizational account and workspaces' roles' permissions are enabled for each of the IdP group's existing users the first (or next) time they sign in to your Unstructured account and then choose
this organizational account.

Whenever you add a user to the IdP group, they get the associated account and workspace roles' permissions the next time they
sign in to your Unstructured account and then choose this organizational account.

If a user is already signed in to your Unstructured account and has chosen this organizational account, but they are not getting the permissions they expect, the user should try signing out of your Unstructured account and then signing back in and choosing this organizational account again, to get those permissions.

If you remove a user from the IdP group, the associated account and workspace roles' permissions are revoked for them the next time they sign in to your Unstructured account and then choose this organizational account. An exception to this is if the user was otherwise [added as an individual user to the account](#add-individual-users-to-an-account).

## Add an IdP group to a workspace

This procedure assumes you have already added the IdP group to your Unstructured account. [Learn how](#add-an-idp-group-to-your-unstructured-account).

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace you want to add the IdP group to.

3. In the top navigation bar, in the workspace selector, select the name of the workspace you want to add the IdP group to.

   <Note>
     If the workspace selector is not showing next to the organizational account selector then, in the top navigation bar,
     click the **Workspaces** tab, and then click the name of the workspace you want to add the IdP group to.
   </Note>

4. On the **Members** tab, click **Add New +**, and then click **+ Add Group**.

5. Select the IdP group to add and its [workspace role](/pipelines/account/roles#workspace-roles) for this workspace, and then click **Continue**.

## Change a workspace role for an IdP group

This procedure assumes you have already added the IdP group to your Unstructured account and the workspace within that account. [Learn how](#add-an-idp-group-to-your-unstructured-account).

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace that refers to the IdP group whose role you want to change.

3. In the top navigation bar, in the workspace selector, select the name of the workspace that refers to the IdP group whose role you want to change.

   <Note>
     If the workspace selector is not showing next to the organizational account selector then, in the top navigation bar,
     click the **Workspaces** tab, and then click the name of the workspace that refers to the IdP group whose role you want to change.
   </Note>

4. On the **Members** tab, click the ellipsis (three dots) next to the name of the IdP group whose workspace role you want to change.

5. Click **Edit Permissions**.

6. Select the IdP group's new [workspace role](/pipelines/account/roles#workspace-roles) for this workspace, and then click **Continue**.

## Change an account role for an IdP group

This procedure assumes you have already added the IdP group to your Unstructured account. [Learn how](#add-an-idp-group-to-your-unstructured-account).

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the IdP group whose organizational account role you want to change.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon, and then click **Manage Account**.
4. In the top navigation bar, click **IdP Groups**.
5. Click the name of the IdP group whose organizational account role you want to change.
6. Next to **Account Role**, click the edit (pencil) icon.
7. Select the new [organizational account role](/pipelines/account/roles#organizational-account-roles) for the IdP group whose organizational account role you want to change, and then click the apply (check mark) icon.

## Add individual users to an account

Unstructured recommends that you add IdP groups to an organizational account, instead of adding individual users.
Managing IdP groups can be easier, faster, and less error-prone than managing individual users.

However, if you must add individual users to an organizational account (for example, if you want to give a user
access to an organizational account at a different level than the access that is granted to them through their IdP groups),
you can do so by following these steps.

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account.

3. In the sidebar, above your user icon, click the **Settings** (gear) icon, and then click **Manage Account**.

4. On the **Members** tab, click **New Member +**.

5. In the **Add New Member** dialog, for **Email**, type the email address of each individual user to add.

6. For all of the email addresses that you typed, select the [organizational account role](/pipelines/account/roles#organizational-account-roles) for all of the individual users at once, and then click **Continue**.

   <Warning>
     An organizational account role that is assigned to an individual user always overrides any organizational account role that is assigned to any IdP group to which that user belongs.
     This override happens the next time they sign in to your Unstructured account and then choose this organizational account.
   </Warning>

7. Click **Continue**.

## Add individual users to a workspace

Unstructured recommends that you add IdP groups to your workspaces, instead of adding individual users.
Managing IdP groups can be easier, faster, and less error-prone than managing individual users.

However, if you must add individual users to workspaces (for example, if you want to give a user
access to a workspace at a different level than the access than is granted to them through their IdP groups),
you can do so by following these steps.

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace for the individual users you want to add.

3. In the top navigation bar, in the workspace selector, select the name of the workspace that you to add the individual users to.

   <Note>
     If the workspace selector is not showing next to the organizational account selector then, in the top navigation bar,
     click the **Workspaces** tab, and then click the name of the workspace you want to add the individual users to.
   </Note>

4. On the **Members** tab, click **Add New+**, and then click **+ Add Member**.

5. In the **Add New Member** dialog, for **Email**, type the email address of each individual user to add.

6. For all of the email addresses that you typed, select the [workspace role](/pipelines/account/roles#workspace-roles) for all of the individual users at once, and then click **Continue**.

   <Warning>
     A workspace role that is assigned to an individual user always overrides any workspace role that is assigned to any IdP group to which that user belongs.
     This override happens the next time they sign in to your Unstructured account and then choose this organizational account.
   </Warning>

7. Click **Continue**.

## Change a workspace role for an individual user

This procedure assumes you have already added the individual user to your Unstructured account and the workspace within that account. [Learn how](#add-individual-users-to-your-unstructured-account).

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account.

3. In the top navigation bar, in the workspace selector, select the name of the workspace.

   <Note>
     If the workspace selector is not showing next to the organizational account selector then, in the top navigation bar,
     click the **Workspaces** tab, and then click the name of the workspace that contains the user for whom you want to change the workspace role.
   </Note>

4. On the **Members** tab, next to the user's email, click the ellipsis (three dots), and then click **Edit Permissions**.

5. Select the new [workspace role](/pipelines/account/roles#workspace-roles) for the user, and then click **Continue**.

## Change an account role for an individual user

This procedure assumes you have already added the individual user to your Unstructured account. [Learn how](#add-individual-users-to-your-unstructured-account).

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the user for whom you want to change their organizational account role.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon, and then click **Manage Account**.
4. On the **Members** tab, click the user's email.
5. Next to **Role**, click the edit (pencil) icon.
6. Select the new [organizational account role](/pipelines/account/roles#organizational-account-roles) for the user, and then click the apply (check mark) icon.

## Remove an individual user from a workspace

This procedure assumes you have already added the individual user to your Unstructured account and the workspace within that account. [Learn how](#add-individual-users-to-your-unstructured-account).

<Warning>
  Removing an individual user from a workspace does not necessarily revoke all access to that workspace!

  After you remove an individual user from a workspace, any workspace role that is assigned to any IdP group to which that user belongs will then be applied.
  This happens the next time they sign in to your Unstructured account.
</Warning>

This procedure assumes you have already added the user to your Unstructured account and the workspace within that account. [Learn how](#add-individual-users-to-your-unstructured-account).

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace.

3. In the top navigation bar, in the workspace selector, select the name of the workspace.

   <Note>
     If the workspace selector is not showing next to the organizational account selector then, in the top navigation bar,
     click the **Workspaces** tab, and then click the name of the workspace that contains the user you want to remove.
   </Note>

4. On the **Members** tab, next to the user's email, click the ellipsis (three dots), and then click **Remove Member**.

5. To confirm the removal, click **Continue**.

## Remove an individual user from an account

This procedure assumes you have already added the individual user to your Unstructured account. [Learn how](#add-individual-users-to-your-unstructured-account).

<Warning>
  Removing an individual user from an organizational account does not necessarily revoke all access to that organizational account!

  After you remove an indivdual user from an account, any organizational account role that is assigned to any IdP group to which that user belongs will then be applied.
  This happens the next time they sign in to your Unstructured account.
</Warning>

This procedure assumes you have already assigned the user to the account. [Learn how](#assign-an-account-role-to-an-individual-user).

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon, and then click **Manage Account**.
4. On the **Members** tab, next to the user's email, click the ellipsis (three dots), and then click **Remove Member**.
5. To confirm the removal, click **Continue**.

## Remove an IdP group from a workspace

This procedure assumes you have already added the IdP group to your Unstructured account and the workspace within that account. [Learn how](#add-an-idp-group-to-your-unstructured-account).

<Warning>
  After you remove an IdP group from a workspace, all users in that group will lose access to the workspace,
  unless they were otherwise [added as individual users to the account](#add-individual-users-to-a-workspace).

  If they were not otherwise added as individual users to the workspace, they will lose access to that workspace
  the next time they sign in to your Unstructured account.
</Warning>

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace.

3. In the top navigation bar, in the workspace selector, select the name of the workspace you want to remove the IdP group from.

   <Note>
     If the workspace selector is not showing next to the organizational account selector then, in the top navigation bar,
     click the **Workspaces** tab, and then click the name of workspace that contains the IdP group you want to remove.
   </Note>

4. On the **Members** tab, next to the IdP group's name, click the ellipsis (three dots), and then click **Remove Member**.

5. To confirm the removal, click **Continue**.

## Remove an IdP group from an account

This procedure assumes you have already added the IdP group to your Unstructured account. [Learn how](#add-an-idp-group-to-your-unstructured-account).

<Warning>
  After you remove an IdP group from an organizational account, all users in that group will lose access to the organizational account,
  unless they were otherwise [added as individual users to the account](#add-individual-users-to-an-account).

  If they were not otherwise added as individual users to the organizational account, they will lose access to that organizational account
  the next time they sign in to your Unstructured account.
</Warning>

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon, and then click **Manage Account**.
4. In the top navigation bar, click **IdP Groups**.
5. Click the name of the IdP group you want to remove from the organizational account.
6. In the settings pane, click **Delete Group**.
7. To confirm the removal, click **Continue**.


# IdP overview
Source: https://docs.unstructured.io/business/idp/overview



<Note>
  The following information applies only to dedicated instance and in-VPC deployments of [Unstructured Business](/business/overview).

  IdP integration is not available for Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** accounts.
</Note>

An *identity provider* (IdP) is a service that manages and verifies the digital identities of users.
It authenticates who a user is and provides that information to other systems (known as *service providers*) to control access.
You can connect your organization's IdP to Unstructured so you can manage who has access across all your connected systems from one place.
Instead of having to manually create and manage user accounts and roles within your Unstructured account, Unstructured can use your organization's IdP to determine
things such as:

* Who can sign in to your Unstructured account's organizational accounts.
* Which roles and permissions they should have within your organizational accounts.
* Revoking access to your organizational accounts—for example, when someone leaves your organization.

## Supported IdPs

Unstructured supports IdPs that use any of the following protocols:

* Keycloak OpenID Connect
* OpenID Connect v1.0
* SAML 2.0

## IdP groups

*IdP groups* are collections of users defined within your IdP—for example, an Engineering group, a Marketing group, or an
Administrators group. Unstructured can use your IdP groups to automatically assign roles and permissions within your Unstructured account
at the account level and for each of your account's workspaces.

## Roles

*Roles* are the sets of permissions that Unstructured can assign to your IdP groups—as well as to individual users separately within your Unstructed account, if needed— through
a common security best-practice technique called *role-based access control* (RBAC). Unstructured has two kinds of roles:

* **Account roles**: These roles include **Super Administrator**, **Account Member**, and **Billing Administrator**. They apply at the account level.
* **Workspace roles**: These roles include **Workspace Administrator**, **Developer**, **Operator**, and **Viewer**. They apply to each of your account's workspaces.

[Learn more about these roles](/pipelines/account/roles).

## Getting started

To connect your organization's IdP to your Unstructured deployment, see [Configure an identity provider](/business/idp/configuration).

## Next steps

After connecting your IdP, you can manage which groups and users have access to your Unstructured account's organizational accounts and workspaces. For details, see [Group mappings](/business/idp/group-mappings).


# Unstructured Business
Source: https://docs.unstructured.io/business/overview



Unstructured offers *business* deployments, which allow you to use an Unstructured **Business** account to run [Unstructured Pipelines](/pipelines/overview)
and the [Unstructured API](/api-reference/overview) as follows:

* <Icon icon="people-group" />  Within a **Business SaaS** deployment - Hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="shield-halved" />  Within a **dedicated instance** deployment - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control. To learn more see [dedicated instance](/business/dedicated-instances/overview).
* <Icon icon="shield" />  Within an **in-VPC** deployment - Hosted within your own VPC on your own cloud infrastructure.

Unstructured **Business** accounts allow for multiple users and offer multiple workspaces. The **dedicated instance** and **in-VPC** account deployment types isolate your Unstructured **Business** account
from all other Unstructured accounts. **Business** account deployments also allow for robust customization of Unstructured's features for your unique needs.

* To sign up for Unstructured **Business**, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales).
* To switch from an existing Unstructured **Let's Go** or **Pay-As-You-Go** account to **Business**, email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Unstructured **Business** deployments provide benefits such as the following:

* *Security, privacy, and ownership*: Your organization might have strict data security requirements to keep your data and models within a virtual private cloud (VPC).
* *Compliance and data sovereignty*: Certain industries and locales might have regulatory requirements that require data to be processed and for data and models to be stored in specific cloud provider regions.
* *Customization*: **Business** account deployments allow for more customization and control options over your environments.
* *Testing and development*: **Business** account deployments can be useful for multi-environment testing and development purposes, allowing you to experiment with Unstructured in a testing environment without affecting your production environment.

## Getting started

To get started with a **Business** account deployment, your organization must first sign a **Business** account agreement with Unstructured.
This agreement outlines the terms and conditions for your organization's use of Unstructured.

To begin the **Business** account agreement process, contact your Unstructured sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

After your organization has signed the **Business** account agreement, the next steps are as follows:

* For a **Business SaaS** deployment, you can start using Unstructured by logging in to Unstructured Pipelines at [https://platform.unstructured.io](https://platform.unstructured.io).
* For a **dedicated instance** deployment, a member of the Unstructured technical enablement team will reach out to you with the unique URL to log in to
  Unstructured Pipelines, along with any next steps that you need to take. To learn more see [dedicated instance](/business/dedicated-instances/overview).
* For an **in-VPC** deployment, your organization will work with a member of the Unstructured technical enablement team to deploy Unstructured
  into your own cloud instrastructure. You are encouraged to begin setting up your target environment as soon as possible. To do this,
  see the onboarding requirements for your [AWS](/business/aws/onboarding-in-vpc), [Azure](/business/azure/onboarding-in-vpc), or\
  [GCP](/business/gcp/onboarding-in-vpc) account.

## Questions? Need help?

Contact your Unstructured sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io), and a member of the Unstructured sales or support teams
will get back to you as soon as possible.


# Security and compliance overview
Source: https://docs.unstructured.io/business/security-compliance/overview



This document outlines the security features and compliance posture of the software system.
It is intended to provide customers and stakeholders with a clear understanding of the security
mechanisms in place and the standards to which the system adheres.

## Encryption

**In Transit**

* All inter-service communication within the Kubernetes cluster is encrypted using mutual TLS (mTLS) via Istio service mesh.
* All ingress and egress communication uses TLS 1.2 or higher.

**At Rest**

* Data stored in blob storage is encrypted at rest using cloud-native encryption mechanisms (e.g., Azure Storage Service Encryption, AWS S3 Server-Side Encryption).
* Encryption keys are managed by the underlying cloud provider and rotated according to provider best practices.

## Secrets management

* Secrets are securely managed using the CSI (Container Storage Interface) driver with support for:

  * Azure Key Vault
  * AWS Secrets Manager
  * Google Secret Manager

* Secrets are mounted into pods at runtime and are never stored in plaintext in the cluster.

* Access to secrets is controlled through fine-grained identity and access management (IAM) policies, tied to the workload identity.

* Rotation of secrets is handled externally and reflected immediately via CSI mounts.

## Authentication and authorization

* End-user authentication is managed through Keycloak, which supports integration with external identity providers such as Azure Active Directory and Google Identity.
* Authentication follows the OAuth2 and OpenID Connect standards.
* Role-based access control (RBAC) is applied across the system.
* Roles and Permissions:

  * Super Admin
  * General Developer
  * Billing

## CI/CD and software supply chain security

* Continuous integration and delivery (CI/CD) pipelines include:

  * Static code analysis
  * Automated testing
  * Software Bill of Materials (SBOM) creation
  * Vulnerability scanning using Grype

* All container images are built from hardened, minimal base images and scanned prior to release.

## Logging and monitoring

* Logs are emitted to standard output and error streams (`stdout`/`stderr`) following container logging best practices.
* Logs can be collected by pluggable logging agents (e.g., Elastic Agent) deployed into the Kubernetes cluster.
* Centralized logging enables customers to integrate with their own observability solutions and meet internal audit requirements.
* Metrics are emitted to an Otel Collector which can be configured to ship to various observability solutions.

## Compliance

The system is designed and operated in accordance with industry-recognized security and data protection standards:

* **SOC 2 Type 2**: Controls are implemented and audited for security, availability, and confidentiality.
* **ISO 27001**: Operational processes align with the ISO 27001 framework for information security management.
* **GDPR**: Data handling practices conform to the General Data Protection Regulation (GDPR) for protection of personal data.
* **FedRAMP**: The system meets the security requirements of the Federal Risk and Authorization Management Program (FedRAMP) for U.S. government systems.
* **CMMC 2.0 Level 2**: The system meets the security requirements for Cybersecurity Maturity Model Certification (CMMC) 2.0 Level 2.

## Additional protections

* Each workflow job runs in an isolated, ephemeral Kubernetes namespace to ensure strong runtime separation.
* Namespaced service isolation within the Kubernetes cluster


# Chunking
Source: https://docs.unstructured.io/concepts/chunking



<Info>
  For the Unstructured open source library version of this page, see [Chunking for open source](/open-source/core-functionality/chunking).
</Info>

<iframe title="YouTube video player" />

After partitioning, *chunking* rearranges the resulting document elements into manageable "chunks" to stay within
the limits of an embedding model and to improve retrieval precision. The goal is to retrieve only parts of documents
that contain only the information that is relevant to a user's query. You can specify if and how Unstructured chunks
those elements, based on your intended end use.

During chunking, Unstructured uses a [basic](#basic-chunking-strategy) chunking strategy that attempts to combine two or more consecutive text elements
into each chunk that fits together within the [max characters](#max-characters-setting) setting. To determine the best max characters setting, see the documentation
for the embedding model that you want to use.

You can further control this behavior with [by title](#chunk-by-title-strategy), [by page](#chunk-by-page-strategy), and [by similarity](#chunk-by-similarity-strategy) chunking strategies.
In all cases, Unstructured will only split individual elements if they exceed the specified max characters length.
After chunking, you will have document elements of only the following types:

* `CompositeElement`: Any text element will become a `CompositeElement` after chunking. A composite element can be a
  combination of two or more original text elements that together fit within the max characters setting. It can also be a single
  element that doesn't leave room in the chunk for any others but fits by itself. Or it can be a fragment of an original
  text element that was too big to fit in one chunk and required splitting.
* `Table`: A table element is not combined with other elements, and if it fits within the max characters setting it will remain as is.
* `TableChunk`: Large tables that exceed the max characters setting are split into special `TableChunk` elements.

Here are a few examples:

```json theme={null}
{
    "type": "CompositeElement",
    "element_id": "3e891e70121efac5f8374e4f1d41f25c",
    "text": "PORTABLE DOCUMENT FORMAT (PDF) SPECIFICATIONS...<full-content-redacted-for-brevity>...",
    "metadata": {
      "filetype": "application/pdf",
      "languages": [
        "eng"
      ],
      "page_number": 1,
      "orig_elements": "eJy...<full-content-redacted-for-brevity>...tY0=",
      "filename": "Portable-Document-Format-Specifications.pdf"
    }
}
```

```json theme={null}
{
    "type": "Table",
    "element_id": "2abca97d9f7dde6557cda4731f5e1d01",
    "text": "Date 2005-04-08 2008-06-04 2011-12-20 2012-1-13 2014-9-26 2016-9-14...<full-content-redacted-for-brevity>...",
    "metadata": {
      "text_as_html": "<table><thead><tr><th>Date</th><th>Version</th><th>Summary of Revisions</th></tr></thead><tbody>...<full-content-redacted-for-brevity>...",
      "filetype": "application/pdf",
      "languages": [
        "eng"
      ],
      "page_number": 3,
      "orig_elements": "eJz...<full-content-redacted-for-brevity>...r8=",
      "filename": "Portable-Document-Format-Specifications.pdf"
    }
}
```

```json theme={null}
{
    "type": "TableChunk",
    "element_id": "57602d200f123a317637eb57596ab488",
    "text": "Term Definition Team A team is defined as a social",
    "metadata": {
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "orig_elements": "eJy...<full-content-redacted-for-brevity>...Ys=",
        "page_number": 5
    },
    

},
{
    "type": "TableChunk",
    "element_id": "4fe0a26a9f38c16b1ac2eedb4460db61",
    "text": "unit of more than two people with specified roles",
    "metadata": {
        "filetype": "application/pdf",
        "is_continuation": true,
        "languages": [
            "eng"
        ],
        "orig_elements": "eJy...<full-content-redacted-for-brevity>...w==",
        "page_number": 5
    }
}
```

If the option to include original elements is specified, during chunking the `orig_elements` field is added to the `metadata` field of each chunked element.
The `orig_elements` field is a list of the original elements that were used to create the current chunked element. This list is output in
gzip compressed, Base64-encoded format. To get back to the original content for this list, Base64-decode the list's bytes, and then gzip decompress them as UTF-8.
[Learn how](/api-reference/legacy-api/partition/get-chunked-elements).

After chunking, `Image` elements are not preserved in the output. However,
if High Res partitioning is used and the option to include original elements is also specified, the `orig_elements` field of each chunked element will contain
an `image_base64` field for each detected image and table associated with the original elements listed within `orig_elements`. To get back to the
original content for an `image_base64` field, Base64-decode the field's bytes.
[Learn how](/api-reference/legacy-api/partition/extract-image-block-types).

The following sections provide information about the available chunking strategies and their settings.

<Note>You can change a workflow's preconfigured strategy only through [Custom](/pipelines/workflows#create-a-custom-workflow) workflow settings.</Note>

## Basic chunking strategy

The basic chunking strategy uses only the [max characters](#max-characters-setting) setting (an absolute or "hard" limit) and
[new after n characters](#new-after-n-characters-setting) setting (an approximate or "soft" limit) to combine sequential elements to maximally
fill each chunk.

This strategy adds elements to a chunk until the new after n characters limit is reached. A new chunk is then started.
No chunk will exceed the max characters limit. For elements larger than the "max characters" limit, the text is split into
multiple chunks at spaces or new lines to avoid cutting words.

Table elements are always treated as standalone chunks. If a table is too large, the table is chunked by rows.

This strategy does not use section boundaries, page boundaries, or content similarities to determine
the chunks' contents.

The following diagram illustrates conceptually how a candidate element is chunked to fit within the max characters (hard) and
new after n characters (soft) limits.

* In scenario 1, the candidate element exceeds the hard limit, and so the candidate element will become the first element in the next chunk.
* In scenario 2, the first candidate element exceeds the soft limit but remains within the hard limit. Because the second candidate element begins
  after the soft limit has been reached, the second candidate element will become the first element in the next chunk.
* In scenario 3, the first two candidate elements exceed the soft limit but remain within the hard limit. Even though the third candidate element
  remains within the hard limit, because it begins after the soft limit has been reached, the third candidate element will become the first element in the next chunk.

<img alt="Chunking with hard and soft limits" />

The following two conceptual diagrams show how a basic chunking strategy with a max characters setting of 200 would chunk the following text and table elements.

In this first conceptual diagram, each chunk of text gets as close as possible to the 200-character hard limit without going over, and lexical constructs such as sentence endings are not recognized:

<img alt="Basic chunking of text with a 200-character hard limit" />

In this second conceptual diagram, each chunk for the table also gets as close as possible to the 200-character hard limit without going over. However, for tables,
row endings are also considered in determining chunk boundaries. For this table, the first chunk is close to the 200-character hard limit and also a row ending.
The second chunk is well short of the 200-character hard limit because of a row (and, in this case, also the table) ending:

<img alt="Basic chunking of a table with a 200-character hard limit" />

Context between chunks can be maintained by using the [overlap](#overlap-setting) and [overlap all](#overlap-all-setting) settings.
The overlap setting repeats the specified number of characters from the end of the previous chunk at the beginning of the next chunk.
By default, overlap all is applied only to relatively large elements. If overlap all is set to true, the overlap is applied to all chunks, regardless.

The overlap setting is based on the number of characters, so words might be split.
The overlap setting's character count is included in the chunk size; nonetheless, the chunk's total size must not exceed the max characters setting.

The following conceptual diagram illustrates conceptually how chunks are calculated by setting overlap all to true or false. In this diagram, setting overlap all to true results in a portion at the end of each chunk always being copied over to the beginning of the next chunk.

By default (or by setting overalp all to false), only a portion at the end of Element 6 Part 1 in Chunk 2 being is copied over
to the beginning of Element 6 Part 2 in Chunk 3, because Element 6 is larger than the max characters setting.

<img alt="Chunking with overall all set to true or false" />

The following diagram shows how a basic chunking strategy with a max characters setting of 200, an overlap of 25 characters, and
overlap all set to true would chunk the following text. Note that some of the text is split in the middle of a word:

<img alt="Basic chunking of text with a 200-character hard limit, an overlap of 25 characters, and overlap all set to true" />

To use this chunking strategy, specify **Chunk by character** for a **Chunker** node in a workflow.

## Chunk by title strategy

The by-title chunking strategy attempts to preserve section boundaries when determining the chunks' contents, primarily when
a **Title** element is encountered. The title is used as the section header for the chunk. The max characters and new after n
characters settings are still respected.

The following conceptual diagram illustrates conceptually how elements are chunked when **Title** elements are encountered (see
Chunks 1, 4, and 6), while still respecting the max characters and new after n characters settings (see Chunks 2 and 3):

<img alt="Chunking by title" />

A single chunk should not contain text that occurred in two different sections. When a new section starts, the existing
chunk is closed and a new one is started, even if the next element would fit in the prior chunk.

The following conceptual diagram illustrates this point, in that many **Title** elements can produce many relatively small chunks.

<img alt="Many titles can lead to many chunks by title" />

To reduce the number of chunks, you can use the [combine text under n characters](#combine-text-under-n-characters-setting) setting. This
setting attempts to combine elements into a single chunk until the combine text under n characters limit is reached, as shown in the
following conceptual diagram. In this case, multiple **Title** elements are combined into a single chunk. However, when the
combine text under n characters limit is reached, the chunk is closed and a new one is started. In any case, the new chunk must start with a **Title** element.
For instance, if Element 3 exceeded the combine text under n characters limit, the chunk would be closed and a new one would be started, beginning
with Title 2, followed by Element 3.

<img alt="Chunking with combine text under n characters" />

The following diagram shows how a chunk by title strategy with a max characters setting of 200 would chunk the following text.
Although the first chunk is close to the 200-character hard limit, the second chunk is well short of this limit due to encountering the
title immediately after it, which starts a new chunk:

<img alt="Chunking by title with a 200-character hard limit" />

To use this chunking strategy, specify **Chunk by title** for a **Chunker** node in a workflow.

## Chunk by page strategy

The by-page chunking strategy attempts to preserve page boundaries when determining the chunks' contents.
A single chunk should not contain text that occurred in two different page. When a new page starts, the existing
chunk is closed and a new one is started, even if the next element would fit in the prior chunk.

The following diagram shows how a chunk by page strategy with a max characters setting of 200 would chunk the following text.
Notice that due to the page break, the second chunk is very small, as it could not fit into the first chunk's hard character limit.\
Nonetheless, the second chunk is still part of same page as the first chunk:

<img alt="Chunking by page with a 200-character hard limit" />

To use this chunking strategy, specify **Chunk by page** for a **Chunker** node in a workflow.

## Chunk by similarity strategy

The by-similarity chunking strategy uses the
[sentence-transformers/multi-qa-mpnet-base-dot-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1) embedding model
to identify topically similar sequential elements and combines them into chunks.

As with the other chunking strategies, chunks will never exceed the absolute maximum chunk size set by **Max characters**. For this reason,
not all elements that share a topic will necessarily appear in the same chunk. However, with this strategy you can
guarantee that two elements with low similarity will not be combined in a single chunk.

To use this chunking strategy, specify **Chunk by similarity** for a **Chunker** node in a workflow.

You can control the level of topic similarity you require for elements to have by setting [Similarity threshold](#similarity-threshold).

The following diagram shows how a chunk by similarity strategy with a max characters setting of 1000 and similarity threshold of 0.5 would chunk the following text.
Notice that the two chunks are well short of the 1000-character hard limit, as the paragraph break introduces a convenient lexical construct for
helping determinine the similarities of sentences to each other:

<img alt="Chunking by similarity with a 1000-character hard limit and 0.5 similarity threshold" />

## Max characters setting

Specifies the absolute maximum number of characters in a chunk.

To specify this setting, specify a number for **Max characters**.

This setting applies to all of the chunking strategies.

## Combine text under n characters setting

Combines elements from a section into a chunk until a section reaches a length of this many characters.

To specify this setting, specify a number for **Combine text under n chars**.

This setting applies only to the chunking strategy **Chunk by title**.

## Include original elements setting

If **Include original elements** is enabled, the elements that were used to form a chunk appear in the `metadata` field's `orig_elements` field for that chunk.

This setting applies to all of the chunking strategies.

## Multipage sections setting

If **Multipage sections** is enabled, this allows sections to span multiple pages.

This setting applies only to the chunking strategy **Chunk by title**.

## New after n characters setting

Closes new sections after reaching a length of this many characters. This is an approximate limit.

To specify this setting, specify a number for **New after n characters**.

This setting applies only to the chunking strategies **Chunk by character**, **Chunk by title**, and **Chunk by page**.

## Overlap setting

Applies a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting.

To specify this setting, specify a number for **Overlap** .

This setting applies only to the chunking strategies **Chunk by character**, **Chunk by title**, and **Chunk by page**.

## Overlap all setting

If **Overlap all** box is enabled, applies overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units.

This setting applies only to the chunking strategies **Chunk by character**, **Chunk by title**, and **Chunk by page**.

## Similarity threshold setting

Specifies the minimum similarity that text in consecutive elements must have to be included in the same chunk.
This must be a value between `0.0` and `1.0`, exclusive (`0.01` to `0.99`). The default is `0.5` if not otherwise specified.

To specify this setting, specify a number for **Similarity threshold**.

This setting applies only to the chunking strategy **Chunk by similarity**.

## Contextual chunking

A technique known as *contextual chunking* prepends chunk-specific explanatory context to each chunk.
Contextual chunking has been shown to enhance traditional RAG solutions by yielding
significant improvements in retrieval accuracy, which directly translates to better performance in downstream tasks.
[Contextual retrieval overview from Anthropic](https://www.anthropic.com/news/contextual-retrieval).

To apply contextual chunking, enable **Contextual chunking** in the settings for any chunking strategy.

This chunk-specific explanatory context information is typically a couple of sentences in length.
Contextual chunking happens before any embeddings are generated.

When contextual chunking is applied, the contextual information in each chunk begins with `Prefix:` and ends with a semicolon (`;`).
The chunk's original content begins with `Original:`.

For example, without contextual chunking applied, elements would for instance be generated similar to the following.
Line breaks have been inserted here for readability. The output will not contain these line breaks:

```json theme={null}
{
    "type": "CompositeElement",
    "element_id": "aa482034de5ade41b7223bb3beeb6a22",
    "text": "THE\n\nCONSTITUTION of the United States\n\nG\n\nNATIONAL 
      CONSTITUTION CENTER\n\nWe the People of the United States, in 
      Order to form a more perfect Union, establish Justice, insure 
      ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1
    }
},
{
    "type": "CompositeElement",
    "element_id": "59fbfcfb51e52c426df4c48a620c6031",
    "text": "SECTION. 2\n\nThe House of Representatives shall be 
        composed of Mem- bers chosen every second Year by the People 
        of the several States, and the Electors in each State shall 
        ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 2
    }
},
```

Applying contextual chunking to those same elements would result in the following output.
Line breaks and blank lines have been inserted here for readability. The output will not contain these line breaks and blank lines:

```json theme={null}
{
    "type": "CompositeElement",
    "element_id": "063ed41d2a989191f2281b2d35c4b4ae",
    "text": "Prefix: This is the opening preamble and first section of 
        Article I of the U.S. Constitution, establishing the fundamental 
        purpose of the document and the basic structure of legislative 
        power in Congress. It appears at the very beginning of the main 
        constitutional text, before all other articles and amendments.;
        
        Original: THE\n\nCONSTITUTION of the United States\n\nG\n\nNATIONAL 
        CONSTITUTION CENTER\n\nWe the People of the United States, in 
        Order to form a more perfect Union, establish Justice, insure 
        ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1
    }
},
{
    "type": "CompositeElement",
    "element_id": "2270f6b8c8b4afc668f6277789370ffd",
    "text": "Prefix: This chunk appears in Article I, Section 2 of the 
        U.S. Constitution, which establishes the structure, composition, 
        and powers of the House of Representatives as one of the two 
        chambers of Congress. It follows Section 1's establishment of 
        Congress and precedes Section 3's establishment of the Senate.;
        
        Original: SECTION. 2\n\nThe House of Representatives shall be 
        composed of Mem- bers chosen every second Year by the People 
        of the several States, and the Electors in each State shall
        ...<full-content-redacted-for-brevity>...",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 2
    }
}

```

## Implementing chunking

To have Unstructured perform chunking, do the following:

* For **Unstructured Pipelines** users, add a [Chunker node](/pipelines/workflows#custom-workflow-node-types)
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add a [Chunker node](/api-reference/workflow/nodes/chunker/chunker)
  as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python) whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).

## Learn more

* <Icon icon="blog" />  "Breaking It Down: Chunking Strategies" in [Level Up Your GenAI Apps: Essential Data Preprocessing for Any RAG System](https://unstructured.io/blog/level-up-your-genai-apps-essential-data-preprocessing-for-any-rag-system)
* <Icon icon="blog" />  [Contextual Chunking in Unstructured Platform: Boost Your RAG Retrieval Accuracy](https://unstructured.io/blog/contextual-chunking-in-unstructured-platform-boost-your-rag-retrieval-accuracy)
* <Icon icon="blog" />  [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices).


# Document elements and metadata
Source: https://docs.unstructured.io/concepts/document-elements



<Info>
  For the Unstructured open source library version of this page, see [Document elements and metadata for open source](/open-source/concepts/document-elements).
</Info>

<iframe title="YouTube video player" />

When Unstructured [partitions](/concepts/partitioning) a file, the result is a list of *document elements*, sometimes referred to simply as *elements*. These elements represent different components of the source file.

## Element example

Here's an example of what an element might look like:

```json theme={null}
{
    "type": "NarrativeText",
    "element_id": "5ef1d1117721f0472c1ad825991d7d37",
    "text": "The Unstructured documentation covers the following services:",
    "metadata": {
        "last_modified": "2024-05-01T14:15:22",
        "page_number": 1,
        "languages": ["eng"],
        "parent_id": "56f24319ae258b735cac3ec2a271b1d9",
        "file_directory": "/content",
        "filename": "Unstructured documentation.html",
        "filetype": "text/html"
    }
}
```

Every element has a [type](#element-type); an [element\_id](#element-id); the extracted `text`; and some [metadata](#metadata) which might
vary depending on the element type, file structure, and some additional settings that are applied during
[partitioning](/concepts/partitioning), [chunking](/concepts/chunking), and [enriching](/concepts/enriching/overview). Optionally, the element can also have an
[embeddings](/concepts/embedding) derived from the `text`; the length of `embeddings` depends on the embedding model that is used.

## Element type

Instead of treating all files as strings of plain text, Unstructured preserves the semantic structure of the files.
This gives you more control and flexibility over how you further use the processed files and allows you to take their
structure into consideration. At the same time, normalizing data from various file formats to the Unstructured element
type scheme lets you treat all files the same in your downstream processing, regardless of source format.
For example, if you plan to summarize a file, you might only be interested in the narrative
of the file and not care about its headers and footers. You can easily filter out the elements you don't need by specifying their type.

Here are some examples of the element types your file might contain:

| Element type        | Description                                                                                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Address`           | A text element for capturing physical addresses.                                                                                                     |
| `CodeSnippet`       | A text element for capturing code snippets.                                                                                                          |
| `EmailAddress`      | A text element for capturing email addresses.                                                                                                        |
| `FigureCaption`     | An element for capturing text associated with figure captions.                                                                                       |
| `Footer`            | An element for capturing document footers.                                                                                                           |
| `FormKeysValues`    | An element for capturing key-value pairs in a form.                                                                                                  |
| `Formula`           | An element containing formulas in a file.                                                                                                            |
| `Header`            | An element for capturing document headers.                                                                                                           |
| `Image`             | A text element for capturing image metadata.                                                                                                         |
| `ListItem`          | `ListItem` is a `NarrativeText` element that is part of a list.                                                                                      |
| `NarrativeText`     | `NarrativeText` is an element consisting of multiple, well-formulated sentences. This excludes elements such titles, headers, footers, and captions. |
| `PageBreak`         | An element for capturing page breaks.                                                                                                                |
| `PageNumber`        | An element for capturing page numbers.                                                                                                               |
| `Table`             | An element for capturing tables.                                                                                                                     |
| `Title`             | A text element for capturing titles.                                                                                                                 |
| `UncategorizedText` | Base element for capturing free text from within files. Applies to extracted text not associated with bounding boxes if the input is a PDF file.     |

If you apply chunking, you will also see the `CompositeElement` type.
`CompositeElement` is a chunk formed from text (non-`Table`) elements.
A composite element might be formed by combining one or more sequential elements produced by partitioning. For example,
several individual list items might be combined into a single chunk.

For the most up-to-date list of available element types, see the `TYPE_TO_TEXT_ELEMENT_MAP` type-annotated mapping definition and the
`ElementType` class definition in the [elements.py](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/documents/elements.py)
file, located in the [Unstructured-IO/unstructured](https://github.com/Unstructured-IO/unstructured)
repository in GitHub.

## Element ID

The element ID identifies the specific element. Source connectors generate element ID in one of the following ways, depending on the source connector:

* A SHA-256 hash of the element's text, its position on the page, the page number it's on,
  and the name of the related file. This is to ensure that the ID is deterministic and unique at the file level.
* A [Version 4 UUID](https://docs.python.org/3/library/uuid.html#uuid.uuid4) generated using random numbers.

<Note>
  Older connectors generate SHA-256 hashes for element IDs, while more modern connectors generate UUIDs. Going forward, older connectors will be converted to using UUIDs as well.
</Note>

## Metadata

Unstructured tracks a variety of metadata about the elements extracted from files. Metadata is tracked at the element level within `metadata`.

Element metadata enables you to do things such as:

* Filter file elements based on an element's metadata value. For instance, you might want to limit your scope to elements from a certain page, or you might want to use only elements that have an email matching a regular expression in their metadata.
* Map an element to the page where it occurred so that the original page can be retrieved when that element matches search criteria.

For the most up-to-date list of all available metadata fields, see the
`ElementMetadata` class definition in the [elements.py](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/documents/elements.py)
file, located in the [Unstructured-IO/unstructured](https://github.com/Unstructured-IO/unstructured)
repository in GitHub.

### Common metadata fields

All file types return the following `metadata` fields when the information is available from the source file:

| Metadata field name        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `category_depth`           | The depth of the element relative to other elements of the same category. Category depth is the depth of an element relative to other elements of the same category. It is set by a file partitioner and enables the [document hierarchy](#document-hierarchy) after processing to compute more accurate hierarchies. Category depth might be set using native document hierarchies, for example reflecting `<H1>` or `<H2>` tags within an HTML file or the indentation level of a bulleted list item in a Word document. |
| `coordinates`              | Any X-Y bounding box [coordinates](#element-coordinates).                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `detection_class_prob`     | The detection model class probabilities. Applies only to Unstructured inference using the **High Res** strategy.                                                                                                                                                                                                                                                                                                                                                                                                           |
| `emphasized_text_contents` | The related emphasized text (bold or italic) in the original file.                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `emphasized_text_tags`     | Any tags on the text that are emphasized in the original file.                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `file_directory`           | The related file's directory.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `filename`                 | The related file's filename.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `filetype`                 | The related file's type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `image_base64`             | A Base64 representation of the detected image or table. Only applicable to image and table elements when High Res partitioning is used.                                                                                                                                                                                                                                                                                                                                                                                    |
| `is_continuation`          | True if the element is a continuation of a previous element. Only relevant for chunking, if an element was divided into two due to **Max Characters**.                                                                                                                                                                                                                                                                                                                                                                     |
| `languages`                | Document languages at the file or element level. The list is ordered by probability of being the primary language of the text.                                                                                                                                                                                                                                                                                                                                                                                             |
| `last_modified`            | The related file's last modified date.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `orig_elements`            | For [chunked](/concepts/chunking) elements, a list of the original elements that were used to create the current chunked element.                                                                                                                                                                                                                                                                                                                                                                                          |
| `parent_id`                | The ID of the element's parent element. `parent_id` might be used to infer where an element resides within the overall [document hierarchy](#document-hierarchy). For instance, a `NarrativeText` element might have a `Title` element as a parent (a “subtitle”), which in turn might have another `Title` element as its parent (a "title").                                                                                                                                                                             |
| `text_as_html`             | The HTML representation of the related extracted table. Only applicable to [table elements](#table-specific-metadata).                                                                                                                                                                                                                                                                                                                                                                                                     |

Notes on common metadata fields:

#### Document hierarchy

`parent_id` and `category_depth` enhance hierarchy detection to identify the document
structure in various file formats by measuring relative depth of an element within its category. This is especially
useful in files with native hierarchies like HTML or Word files, where elements like headings or list items inherently define structure.

#### Element coordinates

Some file types support location data for the elements, usually in the form of bounding boxes.

The `coordinates` metadata field contains:

* `points` : These specify the corners of the bounding box starting from the top left corner and proceeding counter-clockwise. The points represent pixels, the origin is in the top left and the `y` coordinate increases in the downward direction.
* `system`: The points have an associated coordinate system. A typical example of a coordinate system is `PixelSpace`, which is used for representing the coordinates of images. The coordinate system has a name, orientation, layout width, and layout height.

### Additional metadata fields by file type

| Field name             | Applicable file types | Description                                                                                                                         |
| ---------------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `attached_to_filename` | MSG                   | The name of the file that the attached file is attached to.                                                                         |
| `bcc_recipient`        | EML                   | The related [email](#email) BCC recipient.                                                                                          |
| `cc_recipient`         | EML                   | The related [email](#email) CC recipient.                                                                                           |
| `email_message_id`     | EML                   | The related [email](#email) message ID.                                                                                             |
| `header_footer_type`   | Word Doc              | The pages that a header or footer applies to in a [Word document](#microsoft-word-files): `primary`, `even_only`, and `first_page`. |
| `image_mime_type`      | HTML, image, PDF      | The MIME type of the image.                                                                                                         |
| `image_url`            | HTML                  | The URL to the image.                                                                                                               |
| `link_start_indexes`   | HTML, PDF             | A list of the index locations within the extracted content where the `links` can be found.                                          |
| `link_texts`           | HTML                  | A list of text strings that are associated with the `link_urls`.                                                                    |
| `link_urls`            | HTML                  | A list of URLs within the extracted content.                                                                                        |
| `links`                | PDF                   | A list of links within the extracted content.                                                                                       |
| `page_name`            | XLSX                  | The related sheet's name in an [Excel file](#microsoft-excel-files).                                                                |
| `page_number`          | DOCX, PDF, PPT, XLSX  | The related file's page number.                                                                                                     |
| `section`              | EPUB                  | The book section title corresponding to a table of contents.                                                                        |
| `sent_from`            | EML                   | The related [email](#email) sender.                                                                                                 |
| `sent_to`              | EML                   | The related [email](#email) recipient.                                                                                              |
| `signature`            | EML                   | The related [email](#email) signature.                                                                                              |
| `subject`              | EML                   | The related [email](#email) subject.                                                                                                |

Here are some notes on additional metadata fields by file type:

#### Email

For emails, metadata will contain the following fields, where available:

* `bcc_recipient`
* `cc_recipient`
* `email_message_id`
* `sent_from`
* `sent_to`
* `signature`
* `subject`

`sent_from` is a list of strings because the [RFC 822](https://www.rfc-editor.org/rfc/rfc822) spec for emails allows for multiple sent from email addresses.

#### Microsoft Excel files

For Excel files, metadata will contain a `page_name` element, which corresponds to the sheet name in the Excel
file.

#### Microsoft Word files

Headers and footers in Word files include a `header_footer_type` indicating which page a header or footer applies to.
Valid values are `"primary"`, `"even_only"`, and `"first_page"`.

### Table-specific metadata

For `Table` elements, the raw text of the table will be stored in the `text` attribute for the element, and HTML representation
of the table will be available in the element metadata under `text_as_html`.
Unstructured will automatically extract all tables for all doc types if you check the **Infer Table Structure** in the **ConnectorSettings** area of the **Transform** section of a workflow.

Here's an example of a table element. The `text` of the element will look like this (line breaks are added here for readability):

```text theme={null}
Dataset Base Model1 Large Model Notes 
PubLayNet [38] F / M M Layouts of modern scientific documents 
PRImA [3] M - Layouts of scanned modern magazines and scientific reports 
Newspaper [17] F - Layouts of scanned US newspapers from the 20th century 
TableBank [18] F F Table region on modern scientific and business document 
HJDataset [31] F / M - Layouts of history Japanese documents
```

And the `text_as_html` metadata for the same element will look like this (line breaks are added here for readability):

```html theme={null}
<table>
    <thead>
        <th>Dataset</th>
        <th>| Base Model’</th>
        <th>| Notes</th>
    </thead>
    <tr>
        <td>PubLayNet</td>
        <td>[38] F/M</td>
        <td>Layouts of modern scientific documents</td>
    </tr>
    <tr>
        <td>PRImA [3]</td>
        <td>M</td>
        <td>Layouts of scanned modern magazines and scientific reports</td>
    </tr>
    <tr>
        <td>Newspaper</td>
        <td>F</td>
        <td>Layouts of scanned US newspapers from the 20th century</td>
    </tr>
    <tr>
        <td>TableBank</td>
        <td>F</td>
        <td>Table region on modern scientific and business document</td>
    </tr>
    <tr>
        <td>HJDataset [31]</td>
        <td>F/M</td>
        <td>Layouts of history Japanese documents</td>
    </tr>
</table>
```

### Data connector metadata fields

Documents can include additional file metadata, based on the specified source connector.

#### Common data connector metadata fields

* `date_created`
* `date_modified`
* `date_processed`
* `permissions_data`
* `record_locator`
* `url`
* `version`

#### Additional metadata fields by connector type (within record\_locator)

| Source connector | Additional metadata                                  |
| ---------------- | ---------------------------------------------------- |
| Astra DB         | `document_id`                                        |
| Confluence       | `document_id`, `space_id`                            |
| Discord          | `channel_id`                                         |
| Dropbox          | `file_id`, `protocol`, `remote_file_path`            |
| Elasticsearch    | `hosts`                                              |
| Google Drive     | `file_id`                                            |
| Jira             | `id`, `key`                                          |
| MongoDB          | `collection`, `database`, `document_id`              |
| Notion           | `database_id` (for databases), `page_id` (for pages) |
| OpenSearch       | `hosts`                                              |
| S3               | `protocol`, `remote_file_path`                       |
| SharePoint       | `server_relative_path`, `user_pname`                 |

#### File permission metadata

Some source connectors output permissions information that they can find in the source location about the processed source files, and associate that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following source connectors output permissions metadata:

* [Box](/api-reference/workflow/sources/box)
* [Confluence](/api-reference/workflow/sources/confluence)
* [Google Drive](\(/api-reference/workflow/sources/google-drive\))
* [IBM FileNet](/api-reference/workflow/sources/filenet)
* [SharePoint](/api-reference/workflow/sources/sharepoint)

For more information on how a particular source connector extracts permissions data, refer to the documentation for that connector.

### VLM generated HTML elements

The vision language model (VLM) [partitioner](/concepts/partitioning) also generates an HTML representation of the Unstructured elements that are produced.
Unstructured has developed an element ontology that assigns incoming Unstructured elements to these various defined element ontology types.
These element ontology types are used to generate standard HTML elements with the element ontology type as class attributes on
those HTML elements. The generated HTML elements
are output as `text_as_html` along with their `parent_id` in `metadata`, to allow for easier HTML reconstruction of the entire document as needed.

For example, given the following table element produced with the VLM partitioner, the `text_as_html` field is an HTML representation of the
derived table, and `parent_id` is the `element_id` of the Unstructured element for the page that contains this table. (Line breaks are added here to the
`text` and `text_as_html` fields for readability.)

```json theme={null}
{
    "type": "Table",
    "element_id": "c60aea37616e3db75660918c6d657c38",
    "text": "ITEM QUANTITY PRICE TOTAL 
        Office Desk (Oak wood, 140x70 cm) 2 $249 $498 
        Ergonomic Chair (Adjustable height & lumbar support) 3 $189 $567 
        Whiteboard Set (Magnetic, 90x60 cm + 4 markers) 2 $59 $118 
        SUBTOTAL $1,183 
        VAT (19%) $224.77 
        TOTAL $1,407.77",
    "metadata": {
        "category_depth": 1,
        "page_number": 1,
        "parent_id": "8cc3b39afcd948d49d85084eaae80ff8",
        "text_as_html": 
            "<table class=\"Table\" id=\"958308a90ccd4fcb825cb12eed20d103\">
                <thead>
                    <tr>
                        <th>ITEM</th>
                        <th>QUANTITY</th>
                        <th>PRICE</th>
                        <th>TOTAL</th>
                    </tr>
                </thead>
                <tbody>
                    <tr>
                        <td>Office Desk (Oak wood, 140x70 cm)</td>
                        <td>2</td>
                        <td>$249</td>
                        <td>$498</td>
                    </tr>
                    <tr>
                        <td>Ergonomic Chair (Adjustable height &amp; lumbar support)</td>
                        <td>3</td>
                        <td>$189</td>
                        <td>$567</td>
                    </tr>
                    <tr>
                        <td>Whiteboard Set (Magnetic, 90x60 cm + 4 markers)</td>
                        <td>2</td>
                        <td>$59</td>
                        <td>$118</td>
                    </tr>
                    <tr>
                        <td colspan=\"3\">SUBTOTAL</td>
                        <td>$1,183</td>
                    </tr>
                    <tr>
                        <td colspan=\"3\">VAT (19%)</td>
                        <td>$224.77</td>
                    </tr>
                    <tr>
                        <td colspan=\"3\">TOTAL</td>
                        <td>$1,407.77</td>
                    </tr>
                </tbody>
            </table>",
        "languages": [
          "eng"
        ],
        "filetype": "application/pdf",
        "partitioner_type": "vlm_partition",
        "filename": "invoice.pdf"
    }
}
```

For the most up-to-date list of available element ontology types, see the
[ontology.py](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/documents/ontology.py)
file, located in the [Unstructured-IO/unstructured](https://github.com/Unstructured-IO/unstructured) repository in GitHub.

For the most up-to-date list of mappings between element ontology types and Unstructured element types, see the
[mappings.py](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/documents/mappings.py)
file, located in the [Unstructured-IO/unstructured](https://github.com/Unstructured-IO/unstructured) repository in GitHub.

To learn more, see [The Case for HTML as the Canonical Representation in Document AI](https://unstructured.io/blog/the-case-for-html-as-the-canonical-representation-in-document-ai).


# Embedding
Source: https://docs.unstructured.io/concepts/embedding



<Info>
  For the Unstructured open source library version of this page, see [Embedding for open source](/open-source/core-functionality/embedding).
</Info>

<iframe title="YouTube video player" />

After partitioning, chunking, and summarizing, the *embedding* step creates arrays of numbers
known as *vectors*, representing the text that is extracted by Unstructured.
These vectors are stored or *embedded* next to the text itself. These vector embeddings are generated by an
embedding model that is provided by
an *embedding provider*.

You typically save these embeddings in a *vector store*.
When a user queries a retrieval-augmented generation (RAG) application, the application can use a vector database to perform
a similarity search in that vector store
and then return the items whose embeddings are the closest to that user's query.

Here is an example of a document element generated by Unstructured, along with its vector embeddings generated by
the embedding model [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
on Hugging Face:

```json theme={null}
{
    "type": "Title",
    "element_id": "fdbf5369-4485-453b-9701-1bb42c83b00b",
    "text": "THE CONSTITUTION of the United States",
    "metadata": {
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "filename": "constitution.pdf",
        "data_source": {
            "record_locator": {
                "path": "/input/constitution.pdf"
            },
            "date_created": "1723069423.0536132",
            "date_modified": "1723069423.055078",
            "date_processed": "1725666244.571788",
            "permissions_data": [
                { 
                    "mode": 33188
                }
            ]
        }
    },
    "embeddings": [
        -0.06138836592435837,
        0.08634615689516068,
        -0.019471267238259315,
        "<full-results-omitted-for-brevity>",
        0.0895417109131813,
        0.05604064092040062,
        0.01376157347112894
    ]
}
```

[Understanding embedding models: make an informed choice for your RAG](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).

## Generate embeddings

To have Unstructured generate embeddings, do the following:

* For **Unstructured Pipelines** users, add an [Embedder node](/pipelines/workflows#custom-workflow-node-types)
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add an [Embedder node](/api-reference/workflow/nodes/embedder/embedder)
  as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python) whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).

When choosing an embedding model, be sure to pay attention to the number of dimensions for the model you choose. This number must match the number of dimensions in the
embeddings field of your destination connector's table, collection, or index.

## Chunk sizing and embedding models

If your workflow has an [Embedder](/concepts/embedding) node, your workflow's [Chunker](/concepts/chunking) node settings must stay within the selected embedding model's token limits.
Exceeding these limits will cause workflow failures.

Set your **Chunker** node's **Max Characters** to a value at or below Unstructured's recommended maximum chunk size for your selected embedding model,
as listed in the following table's last column.

<Note>
  The following list applies only to Unstructured **Let's Go** and **Pay-As-You-Go** accounts.

  For Unstructured **Business** accounts, see your Unstructured account administrator for your list of available embedding models.
  To add more embedding models to your list, contact your Unstructured account administrator or Unstructured sales representative,
  or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Note>

| Embedding model                | Dimensions | Tokens | Chunker Max Characters<sup>\*</sup> |
| ------------------------------ | ---------- | ------ | ----------------------------------- |
| *Amazon Bedrock*               |            |        |                                     |
| Cohere Embed English           | 1024       | 512    | 1792                                |
| Cohere Embed Multilingual      | 1024       | 512    | 1792                                |
| Titan Embeddings G1 - Text     | 1536       | 8192   | 28672                               |
| Titan Multimodal Embeddings G1 | 1024       | 256    | 896                                 |
| Titan Text Embeddings V2       | 1024       | 8192   | 28672                               |
| *Azure OpenAI*                 |            |        |                                     |
| Text Embedding 3 Large         | 3072       | 8192   | 28672                               |
| Text Embedding 3 Small         | 1536       | 8192   | 28672                               |
| Text Embedding Ada 002         | 1536       | 8192   | 28672                               |
| *Together AI*                  |            |        |                                     |
| M2-Bert 80M 32K Retrieval      | 768        | 8192   | 28672                               |
| *Voyage AI*                    |            |        |                                     |
| Voyage 3                       | 1024       | 32000  | 112000                              |
| Voyage 3 Large                 | 1024       | 32000  | 112000                              |
| Voyage 3 Lite                  | 512        | 32000  | 112000                              |
| Voyage Code 2                  | 1536       | 16000  | 56000                               |
| Voyage Code 3                  | 1024       | 32000  | 112000                              |
| Voyage Finance 2               | 1024       | 32000  | 112000                              |
| Voyage Law 2                   | 1024       | 16000  | 56000                               |
| Voyage Multimodal 3            | 1024       | 32000  | 112000                              |

<sup>\*</sup> This is an approximate value, determined by multiplying the embedding model's token limit by 3.5.


# Generative OCR optimization
Source: https://docs.unstructured.io/concepts/enriching/generative-ocr



After partitioning, you can have a vision language model (VLM) optimize the fidelity of text blocks that Unstructured
initially processed during its partitioning phase.

Here are a few examples of Unstructured's output of text blocks that were initially processed, and the more accurate
version of these text blocks that were optimized by using Claude Sonnet 4. Irrelevant lines of output have been omitted here for brevity.

Example 1: Vertical watermarked text

<img alt="Vertical watermarked text example" />

Before (vertical watermarked text, represented incorrectly):

```json theme={null}
{
    "...": "...",
    "text": "3 2 0 2 t c O 9 2 ] V C . s c [ 2 v 9 0 8 6 1 . 0 1 3 2 : v i X r",
    "...": "..."
}
```

After (vertical watermarked text, now represented correctly from the original content):

```json theme={null}
{
    "...": "...",
    "text": "arXiv:2310.16809v2 [cs.CV] 29 Oct 2023",
    "...": "..."
}
```

Example 2: Hyperlink

<img alt="Hyperlinked text example" />

Before (hyperlink, represented incorrectly):

```json theme={null}
{
    "...": "...",
    "text": "con/Yuliang-Liu/MultinodalOCR|",
    "...": "..."
}
```

After (hyperlink, now represented correctly from the original content):

```json theme={null}
{
    "...": "...",
    "text": "https://github.com/Yuliang-Liu/MultimodalOCR",
    "...": "..."
}
```

Example 3: Chinese characters

<img alt="Chinese characters example" />

Before (Chinese characters, represented incorrectly):

```json theme={null}
{
    "...": "...",
    "text": "GT SHE GPT4-V: EHES",
    "...": "..."
}
```

After (Chinese characters, now represented correctly from the original content, expressed as Unicode):

```json theme={null}
{
    "...": "...",
    "text": "GT : \u91d1\u724c\u70e7\u814a GPT4-V: \u6587\u9759\u5019\u9e1f",
    "...": "..."
}
```

## Improve text fidelity with generative OCR

To have Unstructured perform generative OCR optimization, do the following:

* For **Unstructured Pipelines** users, add an [Enrichment node](/pipelines/workflows#custom-workflow-node-types) of type **Generative OCR**
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add a [Generative OCR task](/api-reference/workflow/nodes/enrichment/enrichment-generative-ocr).
  You add this task as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).


# Image descriptions
Source: https://docs.unstructured.io/concepts/enriching/image-descriptions



<iframe title="YouTube video player" />

After partitioning, you can have Unstructured generate text-based summaries of detected images.

This summarization is done by using models offered through various model providers.

Here is an example of the output of a detected image using GPT-4o. Note specifically the `text` field that is added.
In this `text` field, `type` indicates the kind of image that was detected (in this case, a `diagram`), and `description` is a summary of the image.
Line breaks have been inserted here for readability. The output will not contain these line breaks.

<img alt="Example of a diagram" />

```json theme={null}
{
  "type": "Image",
  "element_id": "dd1fb72db7937725c9a781906098e6f8",
  "text": "{\n    
    \"type\": \"diagram\",\n    
    \"description\": \"User uploads a flowchart image via a Web Browser, which is then 
      converted to a Base64 Encoded Image. This image is sent to the Back-end System 
      (Node.js) where it is processed by the AI Model Adapter. The output undergoes 
      Validation and Rendering, resulting in Normalized Mermaid Code. AI Assisted 
      Editing is available through an AI Assistant, which allows for the Regenerated 
      Flowchart Image to be viewed again in the Web Browser.\\n\\n
      Text in the image:\\n
        - User\\n
        - Upload flowchart image\\n
        - Web Browser\\n
        - Base64 Encoded Image\\n
        - Back-end System (Node.js)\\n
        - AI Model Adapter\\n
        - Validation and Rendering\\n
        - Normalized Mermaid Code\\n
        - AI Assisted Editing\\n
        - AI Assistant\\n
        - Regenerated Flowchart Image\"\n
  }",
  "metadata": {
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_base64": "/9j...<full results omitted for brevity>...Q==",
    "image_mime_type": "image/jpeg",
    "filename": "7f239e1d4ef3556cc867a4bd321bbc41.pdf",
    "data_source": {}
  }
}
```

For technical drawings, the `text` field will contain a `type` of `technical drawing`; `description` with `texts` containing text strings found in the drawing,
`tables` containing HTML representations of tables found in the drawing, and a `description` containing a summary of the drawing.
Here is an example. Line breaks have been inserted here for readability. The output will not contain these line breaks.

<img alt="Example of a technical drawing" />

```json theme={null}
{
  "type": "Image",
  "element_id": "7877acdd762f2afc65b193fa89d8ef46",
  "text": "{\n  
    \"type\": \"technical drawing\",\n  
    \"description\": {\n    
      \"texts\": [\n
        \"RTD 1\",\n      
        \"RTD 2\",\n      
        \"01\",\n      
        \"18.50\\\" Cable Length\",\n      
        \"02\",\n      
        \"1/4\\\" Heat Shrink\",\n      
        \"6X Strip wires 0.100\\\" - 0.115\\\" before crimping\",\n      
        \"2X 1.50\",\n      
        \"22.25\\\" Cable Length\"\n    
      ],\n    
      \"tables\": "<table>
        <thead>
          <tr>
            <th>Item</th>
            <th>Quantity</th>
            <th>Part Number</th>
            <th>Description</th>
            <th>Supplier</th>
            <th>Supplier PN</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td>1</td>
            <td>6</td>
            <td>002622</td>
            <td>Conn Socket 20-24AWG Gold</td>
            <td>Digikey</td>
            <td>WM7082CT-ND</td>
          </tr>
          <tr>
            <td>2</td>
            <td>1</td>
            <td>002647</td>
            <td>Conn Recept 16pos 3mm Dual Row</td>
            <td>Digikey</td>
            <td>WM2490-ND</td>
          </tr>
          <tr>
              <td>3</td>
              <td>2</td>
              <td>102961-01</td>
              <td>M12 Q/D Cable, Elbow, 4-Pole, 5m</td>
              <td>Automation Direct</td>
              <td>EVT222</td>
          </tr>
        </tbody>
      </table>",\n    
      \"description\": \"The technical drawing depicts a wiring setup involving two 
          RTDs (Resistance Temperature Detectors) labeled RTD 1 and RTD 2. Each RTD 
          is connected via cables with specified lengths: RTD 1 has an 18.50-inch 
          cable length, and RTD 2 has a 22.25-inch cable length. The drawing 
          includes annotations for stripping wires, indicating that six wires should 
          be stripped to a length between 0.100 inches and 0.115 inches before 
          crimping. There is a section labeled '1/4\\\" Heat Shrink' and a dimension 
          marked '2X 1.50'. The drawing uses numbered circles to reference specific 
          parts or steps in the process.\"\n  
      }\n
  }",
  "metadata": {
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_base64": "/9j...<full results omitted for brevity>...Q==",
    "image_mime_type": "image/jpeg",
    "filename": "Material-Callouts-c4655c0c.pDF",
    "data_source": {}
  }
}
```

For images of data plot graphs (for example bar charts, line charts, pie charts, and scatter plots),
the `text` field will contain a `type` of `graph`; `description` containing a summary of the graph,
`data` containing numerical values that can be used to regenerate the graph, and `texts` containing
the text strings found in the graph.
Here is an example. Line breaks have been inserted here for readability. The output will not contain these line breaks.

<img alt="Example of a data-plot-graph" />

```json theme={null}
{
  "type": "Image",
  "element_id": "322b067931bc59555ae95dbb63ab9b78",
  "text": "{
    \"type\": \"graph\",
    \"description\": \"This is a comprehensive comparison chart showing 6 different 
      metrics across 7 machine learning models (RF, FFNN, LSTM, MHA, GCN, GCN+MHA, 
      MT+GCN+MHA) represented by different colored bars. The metrics include 
      PAVIS-TW-MSE, PAVIS-LS-MSE, PAVIS-EL@ACC 1, Trainable Parameters, Training 
      Time, and Inference Time. The models are categorized into paradigms: SNN (blue), 
      TNN (green), ReNN (orange), TReNN (purple), and MT-TReNN (red). Key trends show 
      RF and FFNN having higher MSE values but faster inference times, while more 
      complex models like MT+GCN+MHA achieve better accuracy but require more parameters 
      and training time.\", 
    \"data\": {
      \"PAVIS-TW-MSE\": {
        \"RF\": 1.222, \"FFNN\": 1.039, \"LSTM\": 0.817, \"MHA\": 0.790, \"GCN\": 0.800, 
          \"GCN+MHA\": 0.768, \"MT+GCN+MHA\": 0.760
      }, 
      \"PAVIS-LS-MSE\": {
        \"RF\": 0.763, \"FFNN\": 0.878, \"LSTM\": 0.649, \"MHA\": 0.611, \"GCN\": 0.596, 
        \"GCN+MHA\": 0.551, \"MT+GCN+MHA\": 0.549
      }, 
      \"PAVIS-EL@ACC 1\": {
        \"RF\": 79.54, \"FFNN\": 77.05, \"LSTM\": 80.69, \"MHA\": 82.10, \"GCN\": 83.39, 
        \"GCN+MHA\": 85.95, \"MT+GCN+MHA\": 85.98
      }, 
      \"Trainable Parameters\": {
        \"RF\": 328050, \"FFNN\": 237790, \"LSTM\": 264516, \"MHA\": 318941, \"GCN\": 221886, 
        \"GCN+MHA\": 324018, \"MT+GCN+MHA\": 99638
      }, 
      \"Training Time\": {
        \"RF\": 17968, \"FFNN\": 1447, \"LSTM\": 1985, \"MHA\": 2502, \"GCN\": 1620, 
        \"GCN+MHA\": 2808, \"MT+GCN+MHA\": 748
      }, 
      \"Inference Time\": {
        \"RF\": 190, \"FFNN\": 35, \"LSTM\": 75, \"MHA\": 79, \"GCN\": 69, 
        \"GCN+MHA\": 108, \"MT+GCN+MHA\": 18
      }
    }, 
    \"texts\": \"PAVIS - TW - MSE (↓) PAVIS - LS - MSE (↓) PAVIS - EL@ACC 1 (↑) Paradigm 
      SNN TNN ReNN TReNN MT-TReNN 1.222 1.039 0.817 0.790 0.800 0.768 0.760 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 0.763 0.878 0.649 0.611 0.596 0.551 0.549 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 79.54 77.05 80.69 82.10 83.39 85.95 85.98 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA Trainable Parameters (↓) Training Time (ms/epoch) (↓) 
      Inference Time (ms) (↓) 328050 237790 264516 318941 221886 324018 99638 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 17968 1447 1985 2502 1620 2808 748 RF FFNN LSTM 
      MHA GCN GCN+MHA MT+GCN+MHA 190 35 75 79 69 108 18 RF FFNN 
      LSTM MHA GCN GCN+MHA MT+GCN+MHA\"
  }",
  "metadata": {
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 17,
    "image_base64": "/9j...<full results omitted for brevity>...//Z",
    "image_mime_type": "image/jpeg",
    "filename": "250713305v1-48da34f2.pdf",
    "data_source": {}
  }
}
```

<Note>
  The `image_base64` field is generated only for documents or PDF pages that are [partitioned](/concepts/partitioning) by using the High Res strategy. This field is not generated for
  documents or PDF pages that are partitioned by using the Fast or VLM strategy.
</Note>

For workflows that use [chunking](/concepts/chunking), note the following changes:

* Each `Image` element is replaced by a `CompositeElement` element.
* This `CompositeElement` element will contain the image's summary description as part of the element's `text` field.
* This `CompositeElement` element will not contain an `image_base64` field.

Here are three examples of the descriptions for detected images. These descriptions are generated with GPT-4o by OpenAI:

<img alt="Description of an image showing a scatter plot graph" />

<img alt="Description of an image showing the Matthews Correlation Coefficient for different VQA datasets" />

<img alt="Description of an image showing three scatter plots" />

Any embeddings that are produced after these summaries are generated will be based on the `text` field's contents.

## Generate image descriptions

To have Unstructured generate image descriptions, do the following:

* For **Unstructured Pipelines** users, add an [Enrichment node](/pipelines/workflows#custom-workflow-node-types) of type **NER**
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add an [Image Description task](/api-reference/workflow/nodes/enrichment/enrichment-image-description). You add this task
  as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).


# Named entity recognition (NER)
Source: https://docs.unstructured.io/concepts/enriching/ner



<iframe title="YouTube video player" />

After partitioning, you can have Unstructured generate a list of recognized entities and their types (such as the names of organizations, products, and people) in the content, through a process known as *named entity recognition* (NER).
You can also have Unstructured generate a list of relationships between the entities that are recognized.

This NER is done by using models offered through various model providers.

Here is an example of a list of recognized entities and their entity types, along with a list of relationships between those
entities and their relationship types, using GPT-4o. Note specifically the `entities` field that is added to the `metadata` field.

```json theme={null}
{
    "type": "CompositeElement",
    "element_id": "bc8333ea0d374670ff0bd03c6126e70d",
    "text": "SECTION. 3\n\nThe Senate of the United States shall be composed of two Senators from each State, 
        [chosen by the Legislature there- of,]* for six Years; and each Senator shall have one Vote.\n\n
        Immediately after they shall be assembled in Consequence of the first Election, they shall be divided
        as equally as may be into three Classes. The Seats of the Senators of the first Class shall be vacated
        at the Expiration of the second Year, of the second Class at the Expiration of the fourth Year, and of
        the third Class at the Expiration of the sixth Year, so that one third may be chosen every second Year;
        [and if Vacan- cies happen by Resignation, or otherwise, during the Recess of the Legislature of any
        State, the Executive thereof may make temporary Appointments until the next Meeting of the Legislature,
        which shall then fill such Vacancies.]*\n\nC O N S T I T U T I O N O F T H E U N I T E D S T A T E S",
    "metadata": {
        "filename": "constitution.pdf",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 2,
        "entities": {
            "items": [
                {
                    "entity": "Senate",
                    "type": "ORGANIZATION"
                },
                {
                    "entity": "United States",
                    "type": "LOCATION"
                },
                {
                    "entity": "Senator",
                    "type": "ROLE"
                },
                {
                    "entity": "State",
                    "type": "LOCATION"
                },
                {
                    "entity": "Legislature",
                    "type": "ORGANIZATION"
                },
                {
                    "entity": "Executive",
                    "type": "ROLE"
                },
                {
                    "entity": "C O N S T I T U T I O N O F T H E U N I T E D S T A T E S",
                    "type": "DOCUMENT"
                }
            ],
            "relationships": [
                {
                    "from": "Senate",
                    "relationship": "based_in",
                    "to": "United States"
                },
                {
                    "from": "Senator",
                    "relationship": "has_role",
                    "to": "Senate"
                },
                {
                    "from": "Legislature",
                    "relationship": "has_office_in",
                    "to": "State"
                },
                {
                    "from": "Executive",
                    "relationship": "has_role",
                    "to": "State"
                },
                {
                    "from": "C O N S T I T U T I O N O F T H E U N I T E D S T A T E S",
                    "relationship": "dated",
                    "to": "DATE"
                }
            ]
        }
    }
}
```

Here is another example of some of the entities, their entity types, and relationships that are recognized for a given paragraph of text. This information
is generated by GPT-4o by OpenAI:

<img alt="Named entity recognition for information in a paragraph of text" />

By default, the following entity types are supported for NER:

* `PERSON`
* `ORGANIZATION`
* `LOCATION`
* `DATE`
* `TIME`
* `EVENT`
* `MONEY`
* `PERCENT`
* `FACILITY`
* `PRODUCT`
* `ROLE`
* `DOCUMENT`
* `DATASET`

By default, the following entity relationships are supported for NER:

* `PERSON` - `ORGANIZATION`: `works_for`, `affiliated_with`, `founded`
* `PERSON` - `LOCATION`: `born_in`, `lives_in`, `traveled_to`
* `ORGANIZATION` - `LOCATION`: `based_in`, `has_office_in`
* Entity - `DATE`: `occurred_on`, `founded_on`, `died_on`, `published_in`
* `PERSON` - `PERSON`: `married_to`, `parent_of`, `colleague_of`
* `PRODUCT` - `ORGANIZATION`: `developed_by`, `owned_by`
* `EVENT` - `LOCATION`: `held_in`, `occurred_in`
* Entity - `ROLE`: `has_title`, `acts_as`, `has_role`
* `DATASET` - `PERSON`: `mentions`
* `DATASET` - `DOCUMENT`: `located_in`
* `PERSON` - `DATASET`: `published`
* `DOCUMENT` - `DOCUMENT`: `referenced_in`, `contains`
* `DOCUMENT` - `DATE`: `dated`
* `PERSON` - `DOCUMENT`: `published`

You can add, rename, or delete items in this list of default entity types and default entity relationship types.
You can also add any clarifying instructions to the
prompt that is used to run NER. To do this, see the next section.

# Generate a list of entities and their relationships

To have Unstructured generate list of entities and their relationships, do the following:

* For **Unstructured Pipelines** users, add an [Enrichment node](/pipelines/workflows#custom-workflow-node-types) of type **Image Description**
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add a [Named Entity Recognition (NER) task](/api-reference/workflow/nodes/enrichment/enrichment-named-entity-recognition-ner). You add this task
  as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).


# Overview
Source: https://docs.unstructured.io/concepts/enriching/overview



<Info>
  The Unstructured open source library does not support enriching.
</Info>

<iframe title="YouTube video player" />

*Enriching* adds enhancements to the processed data that Unstructured produces. These enrichments include:

* Providing a summarized description of the contents of a detected image. See [Image descriptions](/concepts/enriching/image-descriptions).
* Providing a summarized description of the contents of a detected table. See [Table descriptions](/concepts/enriching/table-descriptions).
* Providing a representation of a detected table in HTML markup format. See [Tables to HTML](/concepts/enriching/table-to-html).
* Providing a list of recognized entities and their types, through a process known as *named entity recognition* (NER). See [Named entity recognition](/concepts/enriching/ner).
* Having a vision language model (VLM) use advanced optical character recognition (OCR) to improve the accuracy of initially-processed text blocks. See [Generative OCR](/concepts/enriching/generative-ocr).

To add an enrichment, add one of the following enrichment node types to an Unstructured workflow:

* [Image Description](/concepts/enriching/image-descriptions) to provide a summarized description of the contents of each detected image.
* [Table Description](/concepts/enriching/table-descriptions) to provide a summarized description of the contents of each detected table.
* [Table to HTML](/concepts/enriching/table-to-html) to provide a representation of each detected table in HTML markup format.00
* [NER](/concepts/enriching/ner) to provide a list of recognized entities and their types by using a technique called *named entity recognition* (NER).
* [Generative OCR](/concepts/enriching/generative-ocr) to have a VLM use advanced OCR to improve the accuracy of initially-processed text blocks.

For multiple enrichments, add an enrichment node for each additional enrichment type to your workflow.

<Warning>
  Unstructured can potentially generate image summary descriptions, table summary descriptions, table-to-HTML output, and generative OCR optimizations, only for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is needed (or allowed).

  Even with these configurations, Unstructured actually generates image summary descriptions, table summary descriptions, and table-to-HTML output only for files that contain images or tables and are also eligible
  for processing with the following partitioning strategies:

  * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
  * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

  Unstructured never generates image summary descriptions, table summary descriptions, or table-to-HTML output for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Fast** partitioning strategy.
  * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images or tables.

  Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
</Warning>


# Table descriptions
Source: https://docs.unstructured.io/concepts/enriching/table-descriptions



<iframe title="YouTube video player" />

After partitioning, you can have Unstructured generate text-based summaries of detected tables.

This summarization is done by using models offered through various model providers.

Here is an example of the output of a detected table using GPT-4o. Note specifically the `text` field that is added.
Line breaks have been inserted here for readability. The output will not contain these line breaks.

```json theme={null}
{
    "type": "Table",
    "element_id": "5713c0e90194ac7f0f2c60dd614bd24d",
    "text": "The table consists of 6 rows and 7 columns. The columns represent 
        inhibitor concentration (g), bc (V/dec), ba (V/dec), Ecorr (V), icorr 
        (A/cm\u00b2), polarization resistance (\u03a9), and corrosion rate 
        (mm/year). As the inhibitor concentration increases, the corrosion 
        rate generally decreases, indicating the effectiveness of the 
        inhibitor. Notably, the polarization resistance increases with higher 
        inhibitor concentrations, peaking at 6 grams before slightly 
        decreasing. This suggests that the inhibitor is most effective at 
        6 grams, significantly reducing the corrosion rate and increasing 
        polarization resistance. The data provides valuable insights into the 
        optimal concentration of the inhibitor for corrosion prevention.",
    "metadata": {
        "text_as_html": "<table>...<full results omitted for brevity>...</table>",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "image_base64": "/9j...<full results omitted for brevity>...//Z",
        "image_mime_type": "image/jpeg",
        "filename": "7f239e1d4ef3556cc867a4bd321bbc41.pdf",
        "data_source": {}
    }
}
```

<Note>
  The `image_base64` field is generated only for documents or PDF pages that are [partitioned](/concepts/partitioning) by using the High Res strategy. This field is not generated for
  documents or PDF pages that are partitioned by using the Fast or VLM strategy.
</Note>

Here are two examples of the descriptions for detected tables. These descriptions are generated with GPT-4o by OpenAI:

<img alt="Description of a table with information about endoscopic datasets" />

<img alt="Description of a table with information about potentiodynamic polarization of stainless steel" />

The generated table's summary will overwrite any text that Unstructured had previously extracted from that table into the `text` field.
The table's original content is available in the `image_base64` field.

<Note>
  The `image_base64` field is generated only for documents or PDF pages that are [partitioned](/concepts/partitioning) by using the High Res strategy. This field is not generated for
  documents or PDF pages that are partitioned by using the Fast or VLM strategy.
</Note>

For workflows that use [chunking](/concepts/chunking), note the following changes:

* If a `Table` element must be chunked, the `Table` element is replaced by a set of related `TableChunk` elements.
* Each of these `TableChunk` elements will contain a summary description only for its own element, as part of the element's `text` field.
* These `TableChunk` elements will not contain an `image_base64` field.

Any embeddings that are produced after these summaries are generated will be based on the new `text` field's contents.

## Generate table descriptions

To have Unstructured generate image descriptions, do the following:

* For **Unstructured Pipelines** users, add an [Enrichment node](/pipelines/workflows#custom-workflow-node-types) of type **Image Description**
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add a [Table Description task](/api-reference/workflow/nodes/enrichment/enrichment-table-description). You add this task
  as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).

## Learn more

* <Icon icon="video" />  [How to Extract Data from Complex Tables](https://unstructured.io/events/how-to-extract-data-from-complex-tables)


# Tables to HTML
Source: https://docs.unstructured.io/concepts/enriching/table-to-html



<iframe title="YouTube video player" />

After partitioning, you can have Unstructured generate representations of each detected table in HTML markup format.

This table-to-HTML output is generated by using agentic AI or a vision language model (VLM).
The agentic AI option typically provides more accurate table-to-HTML output than the VLM option.

Here is an example of the HTML markup output of a detected table using GPT-4o. Note specifically the `text_as_html` field that is added.
Line breaks have been inserted here for readability. The output will not contain these line breaks.

````json theme={null}
{
    "type": "Table",
    "element_id": "31aa654088742f1388d46ea9c8878272",
    "text": "Inhibitor Polarization Corrosion be (V/dec) ba (V/dec) Ecorr (V) icorr 
        (AJcm?) concentration (g) resistance (Q) rate (mmj/year) 0.0335 0.0409 
        \u20140.9393 0.0003 24.0910 2.8163 1.9460 0.0596 .8276 0.0002 121.440 
        1.5054 0.0163 0.2369 .8825 0.0001 42121 0.9476 s NO 03233 0.0540 
        \u20140.8027 5.39E-05 373.180 0.4318 0.1240 0.0556 .5896 5.46E-05 
        305.650 0.3772 = 5 0.0382 0.0086 .5356 1.24E-05 246.080 0.0919",
    "metadata": {
        "text_as_html": "```html\n
            <table>\n
                <tr>\n<th>Inhibitor concentration (g)</th>\n
                    <th>bc (V/dec)</th>\n<th>ba (V/dec)</th>\n<th>Ecorr (V)</th>\n
                    <th>icorr (A/cm\u00b2)</th>\n<th>Polarization resistance (\u03a9)</th>\n
                    <th>Corrosion rate (mm/year)</th>\n
                </tr>\n  
                <tr>\n
                    <td>0</td>\n<td>0.0335</td>\n<td>0.0409</td>\n<td>\u22120.9393</td>\n
                    <td>0.0003</td>\n<td>24.0910</td>\n<td>2.8163</td>\n  
                </tr>\n
                <tr>\n   
                    <td>2</td>\n<td>1.9460</td>\n<td>0.0596</td>\n<td>\u22120.8276</td>\n<td>0.0002</td>\n<td>121.440</td>\n<td>1.5054</td>\n  
                </tr>\n
                <tr>\n
                    <td>4</td>\n<td>0.0163</td>\n<td>0.2369</td>\n<td>\u22120.8825</td>\n<td>0.0001</td>\n<td>42.121</td>\n<td>0.9476</td>\n  
                </tr>\n  
                <tr>\n
                    <td>6</td>\n<td>0.3233</td>\n<td>0.0540</td>\n<td>\u22120.8027</td>\n<td>5.39E-05</td>\n<td>373.180</td>\n<td>0.4318</td>\n  
                </tr>\n  
                <tr>\n
                    <td>8</td>\n<td>0.1240</td>\n<td>0.0556</td>\n<td>\u22120.5896</td>\n<td>5.46E-05</td>\n<td>305.650</td>\n<td>0.3772</td>\n  
                </tr>\n  
                <tr>\n
                    <td>10</td>\n<td>0.0382</td>\n<td>0.0086</td>\n<td>\u22120.5356</td>\n<td>1.24E-05</td>\n<td>246.080</td>\n<td>0.0919</td>\n
                </tr>\n
            </table>\n```",
        "filetype": "application/pdf",
        "languages": [
            "eng"
        ],
        "page_number": 1,
        "image_base64": "/9j...<full results omitted for brevity>...//Z",
        "image_mime_type": "image/jpeg",
        "filename": "embedded-images-tables.pdf",
        "data_source": {}
    }
}
````

<Note>
  The `image_base64` field is generated only for documents or PDF pages that are [partitioned](/concepts/partitioning) by using the High Res strategy. This field is not generated for
  documents or PDF pages that are partitioned by using the Fast or VLM strategy.
</Note>

For workflows that use [chunking](/concepts/chunking), note the following changes:

* If a `Table` element must be chunked, the `Table` element is replaced by a set of related `TableChunk` elements.
* Each of these `TableChunk` elements will contain HTML table output for only its own element.
* None of these `TableChunk` elements will contain an `image_base64` field.

## Generate table-to-HTML output

To have Unstructured generate table-to-HTML output, do the following:

* For **Unstructured Pipelines** users, add an [Enrichment node](/pipelines/workflows#custom-workflow-node-types) of type **Table to HTML**
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add a [Table to HTML task](/api-reference/workflow/nodes/enrichment/enrichment-table-to-html).
  You add this task as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collections applies whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).

## Learn more

* <Icon icon="video" />  [How to Extract Data from Complex Tables](https://unstructured.io/events/how-to-extract-data-from-complex-tables)
* <Icon icon="blog" />  [The Case for HTML as the Canonical Representation in Document AI](https://unstructured.io/blog/the-case-for-html-as-the-canonical-representation-in-document-ai)
* <Icon icon="blog" />  [Preserving Table Structure for Better Retrieval](https://unstructured.io/blog/preserving-table-structure-for-better-retrieval)


# More examples
Source: https://docs.unstructured.io/concepts/examples



This page contains more examples of Unstructured output.

## Doctor's notes

These examples were produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. Personally identifiable
information (PII) has been blurred out of these screenshots.

<img alt="Doctor's note example 1" />

<img alt="Doctor's note example 2" />

<img alt="Doctor's note example 3" />

## Court document

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. Personally identifiable
information (PII) has been blurred out of this screenshot.

<img alt="Court notes" />

## Type-filled form

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI.
The typed information is intended to be fictitious.
A box and arrow has been overlaid onto the screenshot to highlight some of the content that was detected.

<img alt="Handwritten form" />

## Handwriting-filled form

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI.
The handwritten information is intended to be fictitious.

<img alt="Handwritten form" />

## Utility bill

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. While the utility company information is real,
the billing account information is intended to be fictitious. Boxes and arrows have been overlaid onto the screenshot to highlight some of the content that was detected.

<img alt="Utility bill" />

## Product manual pages

These examples were produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. A company logo and name has
been blurred out of one of these screenshots. Boxes and arrows have been overlaid onto some of these screenshots to highlight some of the content that was detected.

<img alt="Product manual" />

<img alt="Multilanguage warnings" />

<img alt="Parts list" />

<img alt="Bookcase instructions" />

## Parts sheets

These examples were produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. Company logos and names have
been blurred out of these screenshots. Boxes and arrows have been overlaid onto these screenshots to highlight some of the content that was detected.

<img alt="Parts sheet for item 70-200" />

<img alt="Parts sheet for item 163T-PRLF" />

## Shipping label

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. The sender and receipient details are meant
to be fictitious.

<img alt="Shipping label" />

## Safety card

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI.

<img alt="Safety card" />

## Product packaging

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. Because this image was
captured on a curve, it is difficult to blur out all identifiable company information without significantly compromising
some of the example's content to be shown. Boxes have been overlaid onto the screenshot to highlight some of the content that was detected.

<img alt="Oatmeal box" />

## Restaurant menu

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. Boxes and arrows have been overlaid onto the screenshot to highlight some of the content that was detected.

<img alt="Restaurant menu" />

## Recipe card

This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI. Boxes and arrows have been overlaid onto the screenshot to highlight some of the content that was detected.

<img alt="Recipe card" />

## Poorly captured image

This image was intentionally captured with poor focus. The captured sign was covered with a thin layer of dirt and
debris. Some graffiti has been blurred out of the top of this image. This example was produced by using the **VLM** partitioning strategy with GPT-4o by OpenAI.
Boxes have been overlaid onto the screenshot to highlight some of the content that was detected.

<img alt="Interpretive sign" />


# Unstructured concepts overview
Source: https://docs.unstructured.io/concepts/overview

Learn how Unstructured processes documents: partitioning, chunking, enriching, embedding, and structured data extraction.

## Document elements and metadata

*Document elements* are the structured pieces of content Unstructured extracts from a file: a title, a paragraph, a table, an image, and so on. Each element also carries *metadata* that describes its origin and processing context.

Preserving semantic structure lets you filter, route, and transform content by type, rather than treating a document as a single blob of text. Metadata gives your downstream applications the context they need to trace results back to their source.

[Learn about document elements and metadata](/concepts/document-elements).

## Partitioning

*Partitioning* is the first processing step. It reads a raw file and extracts its content as structured document elements.

Unstructured supports multiple partitioning strategies, from fast, rule-based text extraction to model-based vision language model (VLM) processing for complex layouts, handwriting, and multilanguage content. Choosing the right strategy lets you balance speed, cost, and output quality for your use case.

[Learn about partitioning](/concepts/partitioning).

## Structured data extraction

The *structured data extractor* lets you define a target JSON schema and have Unstructured extract values from your documents directly into that shape.

This is especially useful for recurring document types such as forms, invoices, or contracts, where you need specific fields in a consistent, application-ready format.

[Learn about structured data extraction](/concepts/structured-data-extractor/data-extractor).

## Chunking

*Chunking* reorganizes partitioned elements into manageable pieces sized for embedding models and optimized for retrieval precision. Instead of embedding entire documents, chunking ensures that each piece of retrieved content is focused and relevant to a user's query.

Unstructured offers several chunking strategies (by character count, section, page, or semantic similarity) so you can tune chunk boundaries to match your content and retrieval goals.

[Learn about chunking](/concepts/chunking).

## Enriching

*Enriching* adds AI-generated enhancements to processed elements. Enrichments include image descriptions, table descriptions, table-to-HTML conversion, named entity recognition (NER), and generative OCR for improving text accuracy in complex documents.

Enrichment gives your downstream applications richer, more useful data from content that would otherwise be hard to work with, such as images, handwritten text, or dense tables.

[Learn about enriching](/concepts/enriching/overview).

## Embedding

*Embedding* converts extracted text into numeric vectors using an embedding model. These vectors capture semantic meaning and are stored alongside the text, ready to be loaded into a vector store.

Vector embeddings power similarity search in RAG applications. When a user submits a query, the application finds the chunks whose embeddings are closest to that query and returns the most relevant results.

[Learn about embedding](/concepts/embedding).

## Examples

For real-world examples of Unstructured output across a range of document types, see [More examples](/concepts/examples).


# Partitioning
Source: https://docs.unstructured.io/concepts/partitioning



<Info>
  For the Unstructured open source library version of this page, see [Partitioning strategies for open source](/open-source/concepts/partitioning-strategies).
</Info>

<iframe title="YouTube video player" />

*Partitioning* extracts content from raw unstructured files and outputs that content as structured [document elements](/concepts/document-elements).

For specific file types, such as image files and PDF files, Unstructured offers special strategies to partition them. Each of these
strategies has trade-offs for output speed, cost to output, and quality of output.

PDF files, for example, vary in quality and complexity. In simple cases, traditional natural language processing (NLP) extraction techniques might
be enough to extract all the text out of a document. In other cases, advanced image-to-text models are required
to process a PDF file. Some of these strategies implement rule-based workflows, which can be faster and cheaper, because they always
extract in the same way, but you might sometimes get lower-quality resolution. Other strategies implement
model-based workflows, which can be slower and costlier because they require a model that performs inference, but you can get higher-quality resolution.
When you choose a partitioning strategy for your files, you should be mindful of these speed, cost, and quality trade-offs.
For example, the **Fast** strategy can be about 100 times faster than leading image-to-text models.

To choose one of these strategies, select one of the following four **Partition Strategy** options for the **Partitioner** node of a workflow.

Unstructured recommends that you choose the **Auto** partitioning strategy in most cases. With **Auto**, Unstructured does all
the heavy lifting, optimizing at runtime for the highest quality at the lowest cost page-by-page.

You should consider the following additional strategies only if you are absolutely sure that your documents are of the same
type. Each of the following strategies are best suited for specific situations. Choosing one of these
strategies other than **Auto** for sets of documents of different types could produce undesirable results,
including reduction in transformation quality.

* **VLM**: For the highest-quality transformation of these file types: `.bmp`, `.gif`, `.heic`, `.jpeg`, `.jpg`, `.pdf`, `.png`, `.tiff`, and `.webp`.
* **High Res**: For all other [supported file types](/pipelines/supported-file-types), and for the generation of bounding box coordinates.
* **Fast**: For text-only documents.

The **Auto** partitioning strategy routes each file as a complete unit to the appropriate partitioning strategy (**VLM**, **High Res**, or **Fast**)
based on the preceding file types. Additionally, for `.pdf` files, the **Auto** partitioning strategy routes these files' pages
on a page-by-page basis, as follows:

* A page is routed to **Fast** when it contains only embedded text and no images or tables are detected.
* All other kinds of pages are routed to **VLM** or **High Res**, depending on the complexity of a page's
  content. Unstructured constantly optimizes its proprietary algorithm for routing to **VLM** or **High Res** in these cases.

## Images and tables in PDF files

The differences between the various partitioning strategies can be more clearly demonstrated by the ways each of these strategies handle images and tables within PDF files.

For example, the **Fast** partitioning strategy skips processing images altogether in PDF files:

<img alt="The Fast strategy skips processing images in PDF files" />

For tables, the **Fast** strategy interprets table cells in PDF files as a mixture of title, list, and uncategorized text elements:

<img alt="The Fast strategy interprets table cells in PDF files as text" />

The **High Res** strategy, by itself, processes images in PDF files sometimes with limited output:

<img alt="The High Res strategy processes images in PDF files with limited results" />

However, when combined with the [image description](/concepts/enriching/image-descriptions) enrichment, the **High Res** strategy can process images in PDF files with better result output:

<img alt="The High Res strategy with image description produces better results" />

For tables, the **High Res** strategy processes tables in PDF files with the table's text and an HTML representation of the table as output:

<img alt="The High Res strategy processes tables in PDF files with the table's text and HTML as output" />

When combined with the [table description](/concepts/enriching/table-descriptions) and [tables to HTML](/concepts/enriching/table-to-html) enrichments, the **High Res** strategy can process tables in PDF files with even richer result output:

<img alt="The High Res strategy with table summarization and table-to-HTML enrichments produces better results" />

The **VLM** strategy processes images in PDF files with image summaries and text as HTML elements as output. The following example shows GPT-4o by OpenAI being used. If
the **Auto** strategy is selected in this example, Unstructured will route to the **VLM** strategy for processing:

<img alt="The VLM strategy processes images in PDF files with image summaries and text as HTML" />

For tables, the **VLM** strategy processes tables in PDF files with the table's text and an HTML representation of the table as output, similar to the **High Res** strategy.
The following example shows GPT-4o by OpenAI being used. If the **Auto** strategy is selected in this example, Unstructured will route to the **VLM** strategy for processing:

<img alt="The VLM strategy processes tables in PDF files with table summaries and text as HTML" />

## Handwriting and multilanguage characters in PDF files

The differences between the various partitioning strategies can be more clearly demonstrated by the ways each of these strategies handle handwriting and multilanguage characters within PDF files.

For example, the **Fast** partitioning strategy skips processing handwriting altogether in PDF files.

The **Fast** strategy processes multilanguage characters in PDF files with limited output, depending on the language. In the following
example, Japanese hiragana characters are processed as text, but the output can be very difficult to work with:

<img alt="The Fast strategy produces cryptic CID codes for hiragana characters" />

For handwriting, the **High Res** strategy typically produces unusable results, for example:

<img alt="The High Res strategy typically produces unusable results for handwriting" />

For multilanguage characters, the **High Res** strategy also typically produces unusable results, for example failing to recognize Japanese hiragana characters:

<img alt="The High Res strategy typically produces unusable results for multilanguage characters" />

The **VLM** strategy can produce great results for handwriting, such as this example that uses GPT-4o by OpenAI:

<img alt="The VLM strategy can process handwriting well" />

The **VLM** strategy also has great support for recognizing multilanguage characters, such as this example that uses GPT-4o by OpenAI to recognize Japanese hiragana characters:

<img alt="The VLM strategy can process Japanese hiragana well" />

## Supported languages

**Fast** partitioning accepts any text inputs, though automatic language detection of those inputs is restricted to [langdetect](https://pypi.org/project/langdetect/).

**High Res** partitioning leverages Tesseract OCR. For the list of languages that Tesseract supports, see:
[Languages/Scripts supported in different versions of Tesseract](https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html).

Language support for **VLM** depends on the model used. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models). The list of supported languages for a particular model is maintained by
that model's provider. For the list of languages that each model supports, see the following, where provided:

* Anthropic:

  * [Claude](https://docs.anthropic.com/en/docs/build-with-claude/multilingual-support)

* OpenAI

  * [GPT](https://help.openai.com/en/articles/8357869-how-to-change-your-language-setting-in-chatgpt#h_513834920e)

* Amazon Bedrock

  * [Claude](https://aws.amazon.com/bedrock/claude/)
  * [Nova](https://aws.amazon.com/ai/generative-ai/nova/)
  * [Llama](https://aws.amazon.com/bedrock/llama/)

* Vertex AI

  * [Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/models#expandable-1)

## Apply a partitioning strategy

To have Unstructured use a specific partitioning strategy, do the following:

* For **Unstructured Pipelines** users, add a [Partitioner node](/pipelines/workflows#custom-workflow-node-types)
  to an Unstructured [custom workflow](/pipelines/workflows#create-a-custom-workflow).
* For **Unstructured API** users, add a [Partitioner node](/api-reference/workflow/nodes/partitioner/partitioner)
  as either as an object in a `workflow_nodes` array
  (for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python) whenever you
  [create a workflow](/api-reference/api/workflow/create-workflow),
  [update a workflow](/api-reference/api/workflow/update-workflow), or
  [create an on-demand workflow job](/api-reference/api/job/create-job).

## Learn more

* <Icon icon="blog" />  [The Case for HTML as the Canonical Representation in Document AI](https://unstructured.io/blog/the-case-for-html-as-the-canonical-representation-in-document-ai)
* <Icon icon="video" />  [How to Extract Data from Complex Tables](https://unstructured.io/events/how-to-extract-data-from-complex-tables)


# Additional examples
Source: https://docs.unstructured.io/concepts/structured-data-extractor/additional-examples



Here are more examples to adapt for your own use. See also the [real estate listing walkthrough](/concepts/structured-data-extractor/data-extractor).

### Caring for houseplants

Using the following image file ([download this file](https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/img/house-plant-care.png)):

<img alt="Caring for houseplants" />

An extraction schema for this file might look like the following:

```json theme={null}
{
  "type": "object",
  "properties": {
    "plants": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "description": "The name of the plant"
          },
          "sunlight": {
            "type": "string",
            "description": "The sunlight requirements for the plant (for example: 'Direct', 'Bright Indirect - Some direct')."
          },
          "water": {
            "type": "string",
            "description": "The watering instructions for the plant (for example: 'Let dry between thorough watering', 'Water when 50-60% dry')."
          },
          "humidity": {
            "type": "string",
            "description": "The humidity requirements for the plant (for example:'Low', 'Medium', 'High')"
          }
        },
        "required": ["name", "sunlight", "water", "humidity"],
        "additionalProperties": false
      }
    }
  },
  "required": ["plants"],
  "additionalProperties": false
}
```

A schema prompt for this file might look like the following:

<Note>
  The schema prompt is available only from the **Start** page.
  The workflow editor does not offer a schema prompt —you must provide an
  extraction schema instead.
</Note>

```text theme={null}
Extract the plant information for each of the plants in this document, and present it in the following format:

- plants: A list of plants.

  - name: The name of the plant.
  - sunlight: The sunlight requirements for the plant (for example: 'Direct', 'Bright Indirect - Some direct').
  - water: The watering instructions for the plant (for example: 'Let dry between thorough watering', 'Water when 50-60% dry').
  - humidity: The humidity requirements for the plant (for example: 'Low', 'Medium', 'High').
```

And Unstructured's elements with extracted data output would look like the following:

```json theme={null}
[
  {
    "type": "DocumentData",
    "element_id": "3be179f1-e1e5-4dde-a66b-9c370b6d23e8",
    "text": "",
    "metadata": {
      "...": "...",
      "extracted_data": {
        "plants": [
          {
            "name": "Krimson Queen",
            "sunlight": "Bright Indirect - Some direct",
            "water": "Let dry between thorough watering",
            "humidity": "Low"
          },
          {
            "name": "Chinese Money Plant",
            "sunlight": "Bright Indirect - Some direct",
            "water": "Let dry between thorough watering",
            "humidity": "Low - Medium"
          },
          {
            "name": "String of Hearts",
            "sunlight": "Direct - Bright Indirect",
            "water": "Let dry between thorough watering",
            "humidity": "Low"
          },
          {
            "name": "Marble Queen",
            "sunlight": "Low- High Indirect",
            "water": "Water when 50 - 80% dry",
            "humidity": "Low - Medium"
          },
          {
            "name": "Sansevieria Whitney",
            "sunlight": "Direct - Low Direct",
            "water": "Let dry between thorough watering",
            "humidity": "Low"
          },
          {
            "name": "Prayer Plant",
            "sunlight": "Medium - Bright Indirect",
            "water": "Keep soil moist",
            "humidity": "Medium - High"
          },
          {
            "name": "Aloe Vera",
            "sunlight": "Direct - Bright Indirect",
            "water": "Water when dry",
            "humidity": "Low"
          },
          {
            "name": "Philodendron Brasil",
            "sunlight": "Bright Indirect - Some direct",
            "water": "Water when 80% dry",
            "humidity": "Low - Medium"
          },
          {
            "name": "Pink Princess",
            "sunlight": "Bright Indirect - Some direct",
            "water": "Water when 50 - 80% dry",
            "humidity": "Medium"
          },
          {
            "name": "Stromanthe Triostar",
            "sunlight": "Bright Indirect",
            "water": "Keep soil moist",
            "humidity": "Medium - High"
          },
          {
            "name": "Rubber Plant",
            "sunlight": "Bright Indirect - Some direct",
            "water": "Let dry between thorough watering",
            "humidity": "Low - Medium"
          },
          {
            "name": "Monstera Deliciosa",
            "sunlight": "Bright Indirect - Some direct",
            "water": "Water when 80% dry",
            "humidity": "Low - Medium"
          }
        ]
      }
    }
  },
  {
    "...": "..."
  }
]
```

### Medical invoicing

Using the following PDF file ([download this file](https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/pdf/spinalogic-bone-growth-stimulator-form.pdf)):

<img alt="Medical invoice" />

An extraction schema for this file might look like the following, which you would then use Unstructured's visual schema builder to define:

```json theme={null}
{
  "type": "object",
  "properties": {
    "patient": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "Full name of the patient."
        },
        "birth_date": {
          "type": "string",
          "description": "Patient's date of birth."
        },
        "sex": {
          "type": "string",
          "enum": ["M", "F", "Other"],
          "description": "Patient's biological sex."
        }
      },
      "required": ["name", "birth_date", "sex"],
      "additionalProperties": false
    },
    "medical_summary": {
      "type": "object",
      "properties": {
        "prior_procedures": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "procedure": {
                "type": "string",
                "description": "Name or type of the medical procedure."
              },
              "date": {
                "type": "string",
                "description": "Date when the procedure was performed."
              },
              "levels": {
                "type": "string",
                "description": "Anatomical levels or location of the procedure."
              }
            },
            "required": ["procedure", "date", "levels"],
            "additionalProperties": false
          },
          "description": "List of prior medical procedures."
        },
        "diagnoses": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "List of medical diagnoses."
        },
        "comorbidities": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "List of comorbid conditions."
        }
      },
      "required": ["prior_procedures", "diagnoses", "comorbidities"],
      "additionalProperties": false
    }
  },
  "required": ["patient", "medical_summary"],
  "additionalProperties": false
}
```

A schema prompt for this file might look like the following:

<Note>
  The schema prompt is available only from the **Start** page.
  The workflow editor does not offer a schema prompt —you must provide an
  extraction schema instead.
</Note>

```text theme={null}
Extract the medical information from this record, and present it in the following format:

- patient

  - name: Full name of the patient.
  - birth_date: Patient's date of birth.
  - sex: Patient's biological sex.

- medical_summary

  - prior_procedures

    - procedure: Name or type of the medical procedure.
    - date: Date when the procedure was performed.
    - levels: Anatomical levels or location of the procedure.

  - diagnoses: List of medical diagnoses.
  - comorbidities: List of comorbid conditions.

Additional extraction instructions:

- name: Extract the full legal name as it appears in the document. Use proper capitalization (for example: "Marissa K. Donovan").
- birth_date: Convert to format "MM/DD/YYYY" (for example: "03/28/1976"),

  - Accept variations: MM/DD/YYYY, MM-DD-YYYY, YYYY-MM-DD, Month DD, YYYY,
  - If only age is given, do not infer birth date - mark as null,

- sex: Extract biological sex as single letter: "M" (Male), "F" (Female), or "X" (Other)

  - Map variations: Male/Man → "M", Female/Woman → "F", Others → "X"

- prior_procedures:

  Extract all surgical and major medical procedures, including:

  - procedure: Use standard medical terminology when possible.
  - date: Format as "MM/DD/YYYY". If only year/month available, use "01" for missing day.
  - levels: Include anatomical locations, vertebral levels, or affected areas.

    - For spine procedures: Use format like "L4 to L5" or "L4-L5".
    - Include laterality when specified (left, right, bilateral).

  - diagnoses:

    Extract all current and historical diagnoses:

    - Include both primary and secondary diagnoses.
    - Preserve medical terminology and ICD-10 descriptions if provided.
    - Include location/region specifications (for example: "radiculopathy — lumbar region").
    - Do not include procedure names unless they represent a diagnostic condition.

  - comorbidities

    Extract all coexisting medical conditions that may impact treatment:

    - Include chronic conditions (for example: "diabetes", "hypertension").
    - Include relevant surgical history that affects current state (for example: Failed Fusion, Multi-Level Fusion).
    - Include structural abnormalities (for example: Spondylolisthesis, Stenosis).
    - Do not duplicate items already listed in primary diagnoses.

Data quality rules:

1. Completeness: Only include fields where data is explicitly stated or clearly indicated.
2. No inference: Do not infer or assume information not present in the source.
3. Preserve specificity: Maintain medical terminology and specificity from source.
4. Handle missing data: Return empty arrays [] for sections with no data, never null.
5. Date validation: Ensure all dates are realistic and properly formatted.
6. Deduplication: Avoid listing the same condition in multiple sections.

Common variations to handle:

- Operative reports: Focus on procedure details, dates, and levels.
- H&P (history & physical): Rich source for all sections.
- Progress notes: May contain updates to diagnoses and new procedures.
- Discharge summaries: Comprehensive source for all data points.
- Consultation notes: Often contain detailed comorbidity lists.
- Spinal levels: C1-C7 (Cervical), T1-T12 (Thoracic), L1-L5 (Lumbar), S1-S5 (Sacral).
- Use "fusion surgery" not "fusion" alone when referring to procedures.
- Preserve specificity: "Type 2 Diabetes" not just "Diabetes" when specified.
- **Multiple procedures same date**: List as separate objects in the array.
- Revised procedures: Include both original and revision as separate entries.
- Bilateral procedures: Note as single procedure with "bilateral" in levels.
- Uncertain dates: If date is approximate (for example, "Spring 2023"), use "01/04/2023" for Spring, "01/07/2023" for Summer, and so on.
- Name variations: Use the most complete version found in the document.
- **Conflicting information**: Use the most recent or most authoritative source.

Output validation:

Before returning the extraction:

1. Verify all required fields are present.
2. Check date formats are consistent.
3. Ensure no duplicate entries within arrays.
4. Confirm sex field contains only "M", "F", or "Other".
5. Validate that procedures have all three required fields.
6. Ensure diagnoses and comorbidities are non-overlapping.
```

And Unstructured's elements with extracted data output would look like the following:

```json theme={null}
[
  {
    "type": "DocumentData",
    "element_id": "e8f09cb1-1439-4e89-af18-b6285aef5d37",
    "text": "",
    "metadata": {
      "...": "...",
      "extracted_data": {
        "patient": {
          "name": "Ms. Daovan",
          "birth_date": "01/01/1974",
          "sex": "F"
        },
        "medical_summary": {
          "prior_procedures": [],
          "diagnoses": [
            "Radiculopathy — lumbar region"
          ],
          "comorbidities": [
            "Diabetes",
            "Multi-Level Fusion",
            "Failed Fusion",
            "Spondylolisthesis"
          ]
        }
      }
    }
  },
  {
    "...": "..."
  }
]
```


# Choose an extraction method: LLM or Regex
Source: https://docs.unstructured.io/concepts/structured-data-extractor/choose-extraction-method



The structured data extractor supports two extraction methods — **LLM** and **Regex** — each suited to different document types and use cases. The extraction **method** setting determines how each field is populated.

## How each extraction method works

Use this table to compare the two methods at a high level — how each processes your documents, the schema format it expects, and what the output looks like.

|                  | **LLM**                                                                                                                                                                       | **Regex**                                                                                                                               |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Choose when      | Values depend on context; you need nested or typed fields                                                                                                                     | Values follow a stable, recognizable pattern (for example: invoice numbers, dates, phone numbers)                                       |
| How it works     | A model reads meaning from text and populates schema-defined fields with inferred values                                                                                      | The extractor scans partitioned text for named patterns and returns matched strings                                                     |
| Schema format    | JSON in [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) format: named fields, types, descriptions, optional nesting | `name` / `pattern` pairs: a label and a regex for each capture field                                                                    |
| Output structure | Typed fields — objects, arrays, numbers, booleans, and strings. See [output examples](/concepts/structured-data-extractor/data-extractor#custom-defined-output).              | An array of matched substrings per pattern name. See [output examples](/concepts/structured-data-extractor/regex-options#regex-output). |
| Model selection  | The provider and model are configurable                                                                                                                                       | No model required — extraction uses a regex engine that matches patterns directly against partitioned text, not a language model        |

<Tip>If both methods could fit, run a small sample with each and compare quality and maintenance cost before you standardize on one.</Tip>

## Available options

The following table shows which options are available for each method. Links go to the detail pages where each option is described.

|                                                                                                        | **LLM**                                                                                    | **Regex**                                                                         |
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| Model selection — choose the LLM provider and model that powers your extraction                        | [Yes](/concepts/structured-data-extractor/llm-options#select-llm-provider-and-model)       | No                                                                                |
| Visual schema builder and JSON upload / export — build your schema visually or import from a JSON file | [Yes](/concepts/structured-data-extractor/llm-options#upload-a-json-file)                  | [Yes](/concepts/structured-data-extractor/regex-options#define-your-schema)       |
| Schema-only output toggle — return extracted fields only, without Unstructured document elements       | [Yes](/concepts/structured-data-extractor/llm-options#schema-only-output-llm)              | [Yes](/concepts/structured-data-extractor/regex-options#schema-only-output-regex) |
| Schema prompt — generate a schema from plain-language instructions                                     | [Yes](/concepts/structured-data-extractor/llm-options#prompt-a-schema)                     | No                                                                                |
| Extraction guidance — instruct the LLM how to format or normalize extracted values                     | [Yes](/concepts/structured-data-extractor/llm-options#extraction-guidance-workflow-editor) | No                                                                                |

## Next steps

**To learn more about the options for each method:**

* [Structured extraction with LLM](/concepts/structured-data-extractor/llm-options) — schema definition, model selection, schema prompt, and extraction guidance
* [Structured extraction with Regex](/concepts/structured-data-extractor/regex-options) — schema definition, validation behavior, output examples, pattern examples, and tools for testing your patterns

**To go straight to step-by-step procedures for using either method:**

* [Using the structured data extractor](/concepts/structured-data-extractor/using-the-structured-data-extractor)


# Structured data extraction
Source: https://docs.unstructured.io/concepts/structured-data-extractor/data-extractor



<Info>
  The Unstructured open source library does not support structured data extraction.
</Info>

<Tip>
  To begin using the structured data extractor right away, skip ahead to the how-to [procedures](/concepts/structured-data-extractor/using-the-structured-data-extractor).
</Tip>

When Unstructured [partitions](/concepts/partitioning) your source documents, the default output is a list of Unstructured [document elements](/concepts/document-elements). Unstructured expresses these elements in its own format — types include `Title`, `NarrativeText`, `UncategorizedText`, `Table`, `Image`, and `List`.

Consider a situation where you have a stack of customer order forms in PDF format — identical layout, but different content per order. You might want to extract common fields like customer IDs, item quantities, and order totals. Normally, that would require custom logic you write yourself.

Unstructured's *structured data extractor* simplifies this kind of scenario without requiring custom logic. It lets you define the target structure up front. Unstructured then extracts values in a consistent JSON format that matches your fields, ready to use in your own applications.

The diagram below shows how data flows from a patient form into JSON output, saved to cloud file storage. From there, you could run a script to insert the JSON as records into a database.

<Frame>
  <img alt="Conceptual flow of structured data extraction" />
</Frame>

To show how the structured data extractor works from a technical perspective, take a look at the following real estate listing PDF. This file is one of the sample files that is available directly from the **Start** page and the workflow editor's **Source** node in Unstructured Pipelines. The file's content is as follows:

<img alt="Sample real estate listing PDF" />

Without the structured data extractor, if you run a workflow that references this file, Unstructured extracts the listing's data in a default format similar to the following (note that the ellipses in this output indicate omitted fields for brevity):

```json theme={null}
[
  {
    "type": "Title",
    "element_id": "3f1ad705648037cf65e4d029d834a0de",
    "text": "HOME FOR FUTURE",
    "metadata": {
      "...": "..."
    }
  },
  {
    "type": "NarrativeText",
    "element_id": "320ca4f48e63d8bcfba56ec54c9be9af",
    "text": "221 Queen Street, Melbourne VIC 3000",
    "metadata": {
      "...": "..."
    }
  },
  {
    "type": "NarrativeText",
    "element_id": "05f648e815e73fe5140f203a62d8a3cc",
    "text": "2,800 sq. ft living space",
    "metadata": {
      "...": "..."
    }
  },
  {
    "type": "NarrativeText",
    "element_id": "27a9ded56b42f559999e48d1dcd76c9e",
    "text": "Recently renovated kitchen",
    "metadata": {
      "...": "..."
    }
  },
  {
    "...": "..."
  }
]
```

In the preceding output, the `text` fields contain information about the listing, such as the street address, the square footage, and one of the listing's features. The default output leaves you parsing values out of generic `text` fields. With the structured data extractor, each value is captured in its own named field — structurally accessible and directly consumable by your applications without additional parsing logic.

<h2>
  Custom-defined output
</h2>

By using the structured data extractor in your Unstructured workflows, you can have Unstructured extract the listing's data in a custom-defined format.

### Elements with extracted data format

The first custom-defined output format is known as the **elements with extracted data** format, as follows (ellipses indicate omitted fields for brevity):

```json theme={null}
[
  {
    "type": "DocumentData",
    "element_id": "f2ee7334-c00a-4fc0-babc-2fcea28c1fb6",
    "text": "",
    "metadata": {
      "...": "...",
      "extracted_data": {
        "street_address": "221 Queen Street, Melbourne VIC 3000",
        "square_footage": 2800,
        "price": 1000000,
        "features": [
          "Recently renovated kitchen",
          "Smart home automation system",
          "2-car garage with storage space",
          "Spacious open-plan layout with natural lighting",
          "Designer kitchen with quartz countertops and built-in appliances",
          "Master suite with walk-in closet and en-suite bath",
          "Covered patio and landscaped backyard garden"
        ],
        "agent_contact": {
          "phone": "+01 555 123456"
        }
      }
    }
  },
  {
    "type": "Title",
    "element_id": "3f1ad705648037cf65e4d029d834a0de",
    "text": "HOME FOR FUTURE",
    "metadata": {
      "...": "..."
    }
  },
  {
    "...": "..."
  }
]
```

In this output, the first element — type `DocumentData` — carries your extracted fields in an `extracted_data` field under `metadata`. From the second element onward, Unstructured also outputs the document's data as its standard document elements and metadata. The following illustrates the layout at a glance:

```text theme={null}
[
  { "type": "DocumentData", ...      ← first element: carries your extracted fields
    "metadata": {
      "extracted_data": { ... }      ← your schema-defined fields live here
    }
  },
  { "type": "Title", ... },          ← second element onward: standard document elements
  { ... }
]
```

### Extracted data only format

The second custom-defined output format is known as the **extracted data only** format, as follows:

```json theme={null}
{
  "street_address": "221 Queen Street, Melbourne VIC 3000",
  "square_footage": 2800,
  "price": 1000000,
  "features": [
    "Recently renovated kitchen",
    "Smart home automation system",
    "2-car garage with storage space",
    "Spacious open-plan layout with natural lighting",
    "Designer kitchen with quartz countertops and built-in appliances",
    "Master suite with walk-in closet and en-suite bath",
    "Covered patio and landscaped backyard garden"
  ],
  "agent_contact": {
    "phone": "+01 555 123456"
  }
}
```

In this format, the document's data is output only as the JSON fields you defined in your schema. The following illustrates what is removed compared to the **elements with extracted data** format, and the resulting structure:

**Elements no longer included:**

```diff theme={null}
- [                                  ← removed: no surrounding array
-   { "type": "DocumentData",        ← removed: no DocumentData element
-     "metadata": {                  ← removed: no metadata wrapper
-       "extracted_data": {          ← removed: no extracted_data nesting
          "street_address": "...",
          "square_footage": ...,
          "features": [ ... ],
          "agent_contact": { ... }
-       }
-     }
-   },
-   { "type": "Title", ... },        ← removed: no standard document elements
- ]
```

**Extracted data only format:**

```text theme={null}
{
  "street_address": "...",    ← fields appear directly at the root
  "square_footage": ...,
  "features": [ ... ],
  "agent_contact": { ... }
}
```

<Note>
  In the workflow editor, the **Schema-Only Output** toggle controls which format is used. Switch it ON to get the **extracted data only** format, or leave it OFF (the default) to get the **elements with extracted data** format.

  * For step-by-step instructions in Unstructured Pipelines, see the [workflow editor procedure](/concepts/structured-data-extractor/using-the-structured-data-extractor#use-the-structured-data-extractor-from-the-workflow-editor).
  * For API users, these correspond to the output formats described in the [Extract node](/api-reference/workflow/workflows#extract-node) API reference.
</Note>

## Limitations

The structured data extractor is not guaranteed to work with the [Pinecone destination connector](/pipelines/destinations/pinecone). This is because Pinecone has strict limits on the amount of metadata that it can manage. These limits are below the threshold of what the structured data extractor typically needs for the amount of metadata that it manages.

## Next steps

* To learn about the two structured data extraction methods — **LLM** and **Regex** — and compare them to decide which fits your use case, see [Choose an extraction method: LLM or Regex](/concepts/structured-data-extractor/choose-extraction-method).
* To go straight to step-by-step procedures, see [Using the structured data extractor](/concepts/structured-data-extractor/using-the-structured-data-extractor).


# Structured Extraction with LLM
Source: https://docs.unstructured.io/concepts/structured-data-extractor/llm-options



When the extraction method is **LLM**, a model reads meaning from your documents and populates schema-defined fields with inferred values. This page covers those options — schema definition, model selection, schema prompt, and extraction guidance. To compare LLM and Regex before choosing, see [Choose an extraction method](/concepts/structured-data-extractor/choose-extraction-method).

* For **Unstructured Pipelines** users, see [Unstructured Pipelines settings for structured extraction with LLM](#unstructured-pipelines-settings-for-structured-extraction-with-llm)
* For **Unstructured API** users, see [Unstructured API Settings for structured extraction with LLM](#unstructured-api-settings-for-structured-extraction-with-llm)

## Unstructured Pipelines settings for structured extraction with LLM

The following sections describe how to use the [Unstructured Pipelines](/pipelines/overview) to specify settings for
structured extraction with LLM.

### Define your schema (Pipelines only)

In Unstructured Pipelines, you can build your extraction schema directly in the visual schema builder, or generate a starting point from a plain-language prompt. Once generated, you can refine the schema in the builder and export it as JSON. Be aware that generating a new schema from the plain-language prompt will overwrite any existing builder content.

<Tip>If you already have a schema in the visual schema builder and want to try generating one from a plain-language prompt, export your current schema to a JSON file first. You can upload it again later if you prefer the original.</Tip>

<h3>
  Visual schema builder and JSON upload/export (Pipelines only)
</h3>

In Unstructured Pipelines, on the **Start** page or in the [workflow editor](/pipelines/workflows#create-a-custom-workflow), you can access the visual schema builder in the **Define Schema** view. From there you can:

* **Upload** a JSON file to the editor.
* **Edit** the fields in the schema directly in the editor.
* **Export** the schema you have defined to a JSON file for reuse.

An extraction schema is a JSON-formatted schema that defines the structure of the data that Unstructured extracts. If you already have an extraction schema defined in a JSON file, you can click **Upload JSON** to upload the file to Unstructured.

<Note>The schema must conform to the [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) guidelines, which are a subset of the [JSON Schema](https://json-schema.org/docs) language. Per OpenAI's guidelines, the maximum supported JSON schema nesting depth is 10 levels.</Note>

The following shows the extraction schema for the sample real estate listing — first in the visual schema builder, then as a JSON schema file.

**The LLM visual schema builder:**

<img alt="LLM visual schema builder showing an extraction schema with the Export schema as JSON option" />

**JSON schema file:**

```json theme={null}
{
  "type": "object",
  "properties": {
    "street_address": {
      "type": "string",
      "description": "The full street address of the property including street number, street name, city, state, and postal code"
    },
    "square_footage": {
      "type": "number",
      "description": "The total living space area of the property, in square feet"
    },
    "price": {
      "type": "number",
      "description": "The listed selling price of the property, in local currency"
    },
    "features": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "A list of property features and highlights"
    },
    "agent_contact": {
      "type": "object",
      "properties": {
        "phone": {
          "type": "string",
          "description": "The agent's contact phone number"
        }
      },
      "required": [
        "phone"
      ],
      "additionalProperties": false,
      "description": "Contact information for the real estate agent"
    }
  },
  "additionalProperties": false,
  "required": [
    "street_address",
    "square_footage",
    "price",
    "features",
    "agent_contact"
  ]
}
```

<h3>
  Plain language in a schema prompt (Pipelines only)
</h3>

Unstructured Pipelines allows you to specify your extraction schema with a schema prompt instead
of by using a visual schema designer or a JSON schema.

A **schema prompt** is plain-language instructions that describe what to extract from your documents, similar to a prompt you would give a chatbot or AI agent. Unstructured *generates* an extraction schema from those instructions: a structured definition (fields, types, and constraints) that guides extraction from the source documents.

<Note>This option is only available from the **Start** page.</Note>

From the **Start** page click **Suggest**, enter your prompt in the **Prompt a Schema** dialog, then click **Generate schema**. Following your prompt instructions, Unstructured will generate a schema that will display in the visual schema builder.

<img alt="Prompt a Schema dialog showing a plain-language prompt for a real estate listing" />

<Warning>Selecting **Generate schema** overwrites the existing schema that's displayed in the **Define Schema** pane.  If you'd like to save the current schema before generating a new one, click the ellipses (three dots) icon, then click **Export schema as JSON**.</Warning>

The generated schema displays in the visual schema builder. You can continue to edit the schema from the visual schema builder if you wish.

For this real estate listing example, you might enter the following prompt:

```text theme={null}
Extract the following information from the listing, and present it in the following format:

- street_address: The full street address of the property including street number, street name, city, state, and postal code.
- square_footage: The total living space area of the property, in square feet.
- price: The listed selling price of the property, in local currency.
- features: A list of property features and highlights.
- agent_contact: Contact information for the real estate agent.
- phone: The agent's contact phone number.
```

The following image shows the generated schema that displays in the visual schema builder.

<img alt="LLM visual schema builder displaying a schema generated from a plain-language prompt" />

<h3>
  Select your LLM provider and model (Pipelines only)
</h3>

In Unstructured Pipelines, you can select a provider and model for the **LLM** extraction method. For **Model**, select your provider and model from the drop-down.

<img alt="Provider and model selection dropdown in the workflow editor Extract node" />

<Note>This option is only available from the [workflow editor](/pipelines/workflows#create-a-custom-workflow).</Note>

### Configure your output (Pipelines only)

In Unstructured Pipelines, once your schema determines which fields to extract and what types they return, settings control what the output looks like. **Schema-only output** lets you strip away Unstructured's document elements and return just the extracted fields. **Extraction guidance** lets you tell the LLM how to format, normalize, or summarize values into the fields your schema defines.

<h3>
  Schema-only output (Pipelines only)
</h3>

In Unstructured Pipelines, the **Schema-Only Output** setting controls whether
Unstructured's document elements are stripped away and returns just the extracted fields.

The **Schema-Only Output** setting applies to both the LLM and Regex extraction methods. In the **workflow editor**, select the workflow’s **Extract** node. Under **Output settings**, you can set **Schema-Only Output** to ON or OFF whenever you edit the workflow.

* When **Schema-Only Output** is ON, the **Extract** node returns only the JSON produced for your explicitly defined fields.

  In workflow JSON, that is the **extracted data only** layout from [Custom defined output](/concepts/structured-data-extractor/data-extractor#custom-defined-output) (no surrounding Unstructured element list).
* When **Schema-Only Output** is OFF (the default), Unstructured also emits the usual document elements and metadata alongside those extracted values.

  In workflow JSON, that is the **elements with extracted data** layout from the same [Custom defined output](/concepts/structured-data-extractor/data-extractor#custom-defined-output) section (structured fields under `DocumentData` plus the rest of the element list).

<img alt="Schema-Only Output toggle in the Extract node Output settings" />

<Note>This option is only available from the **workflow editor**.</Note>

<h3>
  Extraction guidance (Pipelines only)
</h3>

In Unstructured Pipelines, in the [workflow editor](/pipelines/workflows#create-a-custom-workflow), use the **Extraction Guidance Prompt** to tell the LLM how to format, normalize, or present values *after* your schema defines which fields to extract.

<Note>This option is only available from the **workflow editor**.</Note>

The schema still defines *what* to extract (fields, types, and constraints). *Extraction guidance* adds plain-language direction for *how* to format, normalize, or summarize that output when JSON Schema alone is not enough. For example, you can ask the model to standardize addresses, return dates in a consistent format, or summarize long text into a predefined field. You can save this guidance in the **workflow editor** with the **Extract** node settings and with the workflow you're defining, so later runs, including API operations against that workflow, use the same guidance.

You can add or revise an **Extraction Guidance Prompt** in the **workflow editor** after you add or select the **Extract** node. From the **structured data extractor**, click **+ Add Prompt** to enter plain-language instructions for how the LLM should format or present values after your schema has defined the fields. Saving writes the prompt into the node's settings. Extracted values must still conform to the schema; the prompt only describes presentation and cleanup on top of that contract. You can edit and save the extraction guidance again as you iterate.

<img alt="Extraction Guidance Prompt field in the workflow editor Extract node" />

## Unstructured API settings for structured extraction with LLM

The following sections describe how to use the [Unstructured API](/api-reference/overview) to specify settings for
structured extraction with LLM.

<h3>
  Define your schema (API only)
</h3>

An *extraction schema* is a JSON-formatted schema that defines the structure of the data that Unstructured extracts.

<Note>The schema must conform to the [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) guidelines, which are a subset of the [JSON Schema](https://json-schema.org/docs) language. Per OpenAI's guidelines, the maximum supported JSON schema nesting depth is 10 levels.</Note>

To specify an extraction schema with the Unstructured API, use the [LLM method of an Extract node](/api-reference/workflow/nodes/extract/extract-llm).
In this node, set the `schema_to_extract.json_schema` key in the `settings` object
as either as an object in a `workflow_nodes` array
(for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
[create a workflow](/api-reference/api/workflow/create-workflow),
[update a workflow](/api-reference/api/workflow/update-workflow), or
[create an on-demand workflow job](/api-reference/api/job/create-job).

### Specify your LLM provider and model (API only)

You must specify an LLM provider and model for Unstructured to perform the extraction. To do this with the Unstructured API,
use the [LLM method of an Extract node](/api-reference/workflow/nodes/extract/extract-llm).
In this node, set the `provider` and `model` keys in the `settings` object
as either as an object in a `workflow_nodes` array
(for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
[create a workflow](/api-reference/api/workflow/create-workflow),
[update a workflow](/api-reference/api/workflow/update-workflow), or
[create an on-demand workflow job](/api-reference/api/job/create-job).

### Configure your output (API only)

Once your schema determines which fields to extract and what types they return, settings control what the output looks like. **Schema-only output** lets you strip away Unstructured's document elements and return just the extracted fields. **Extraction guidance** lets you tell the LLM how to format, normalize, or summarize values into the fields your schema defines.

### Schema-only output (API only)

You can use the `output_mode` setting with the Unstructured API to control whether
Unstructured's document elements are stripped away and returns just the extracted fields:

* Set `output_mode` to `extracted_data_only` to output only the extracted data as JSON, without any parent `DocumentData` element or any other built-in Unstructured document elements.
* Set `output_mode` to `elements_with_extracted_data` to output the extracted data as JSON, inside of a parent `DocumentData` element. This element is also
  included with any other built-in Unstructured document elements.

To specify this setting, use the [LLM method of an Extract node](/api-reference/workflow/nodes/extract/extract-llm).
In this node, set the `output_mode` key in the `settings` object.
You set this object as either as an object in a `workflow_nodes` array
(for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
[create a workflow](/api-reference/api/workflow/create-workflow),
[update a workflow](/api-reference/api/workflow/update-workflow), or
[create an on-demand workflow job](/api-reference/api/job/create-job).

### Extraction guidance (API only)

You can use the **Extraction Guidance Prompt** setting with the Unstructured API to tell the LLM how to format, normalize, or present values *after* your schema defines which fields to extract.

To specify this setting, use the [LLM method of an Extract node](/api-reference/workflow/nodes/extract/extract-llm).
In this node, set the `schema_to_extract.extraction_guidance` key in the `settings` object
as either as an object in a `workflow_nodes` array
(for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
[create a workflow](/api-reference/api/workflow/create-workflow),
[update a workflow](/api-reference/api/workflow/update-workflow), or
[create an on-demand workflow job](/api-reference/api/job/create-job).

### API limitations

The Unstructured API does not support the following options for structured extraction with LLM. To use either of these options, you must use Unstructured Pipelines instead. To learn how, see the following links:

* [Visual schema builder and JSON upload/export (Pipelines only)](#upload-a-json-file)
* [Plain language in a schema prompt (Pipelines only)](#prompt-a-schema)


# Structured Extraction with Regex
Source: https://docs.unstructured.io/concepts/structured-data-extractor/regex-options



When the extraction method is **Regex**, the structured data extractor scans [partitioned text](/concepts/partitioning) for named patterns and returns an array of matched strings for each field you define. This page covers how to define your schema, understand validation behavior across environments, and interpret the extraction output. To compare Regex and LLM before choosing, see [Choose an extraction method](/concepts/structured-data-extractor/choose-extraction-method).

* For **Unstructured Pipelines** users, see [Unstructured Pipelines settings for structured extraction with Regex](#unstructured-pipelines-settings-for-structured-extraction-with-regex).
* For **Unstructured API** users, see [Unstructured API Settings for structured extraction with Regex](#unstructured-api-settings-for-structured-extraction-with-regex).

## Unstructured Pipelines settings for structured extraction with Regex

The following sections describe how to use the [Unstructured Pipelines](/pipelines/overview) to specify settings for
structured extraction with Regex.

<h3>
  Define your schema (Pipelines only)
</h3>

In Unstructured Pipelines, you can build your extraction schema directly in the visual schema builder, or upload `name` / `pattern` pairs from an existing JSON file.

On the **Start** page or in the **workflow editor**, open the regex **Define Schema** view (**Pattern name** and **Regular expression** rows). From there you can:

* **Upload** a JSON file to load `name` / `pattern` pairs into the form.
* **Edit** `name` / `pattern` rows directly in the editor.
* **Export** the pairs you have defined to a JSON file for reuse.

The following shows an extraction schema for a sample invoice — first in the visual schema builder, then as a JSON file.

**The Regex visual schema builder:**

<img alt="Regex visual schema builder showing name and pattern pairs with the Export schema as JSON option" />

**JSON file (`name` / `pattern` pairs):**

```json theme={null}
[
  {
    "name": "dollar amounts in invoice",
    "pattern": "\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?"
  },
  {
    "name": "dates in invoice",
    "pattern": "\\b\\d{1,2}\\s+(?:JANUARY|FEBRUARY|MARCH|APRIL|MAY|JUNE|JULY|AUGUST|SEPTEMBER|OCTOBER|NOVEMBER|DECEMBER)\\s+\\d{4}\\b"
  }
]
```

#### Escaped and raw formats (Pipelines only)

When you **upload** JSON, Unstructured fills each **Regular expression** field with the **literal** regex for that `pattern`—the same characters you would type by hand. The Pipelines does **not** show the doubled `\` characters that JSON requires inside string values. If you type a pattern by hand, do not enter JSON-style escapes (such as doubled `\`) in that field. Use the table below to compare a raw regex with the same value inside a JSON string.

**Raw regex vs JSON-escaped `pattern` string:**

| Context     | Pattern                                   |
| ----------- | ----------------------------------------- |
| Raw regex   | `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`      |
| JSON string | `\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?` |

When you select **Export schema as JSON**, Unstructured serializes your `name` / `pattern` pairs as JSON and applies the escapes that JSON requires inside string values. For example, a single `\` in the raw pattern becomes `\\` in the saved file.

If you use **Upload JSON**, Unstructured reads that JSON and shows each **Regular expression** as the raw pattern (the same characters you would type by hand), not the JSON-escaped string from the file.

The visual schema builder validates the regex you enter before running the extraction. For more information, see [Regex validation](#regex-validation).

To learn more about regex patterns and see examples, see [Regex pattern examples](#regex-pattern-examples).

<h3>
  Schema-only output (Pipelines only)
</h3>

In Unstructured Pipelines, the **Schema-Only Output** setting controls whether
Unstructured's document elements are stripped away and returns just the extracted fields.

The **Schema-Only Output** setting applies to both the LLM and Regex extraction methods. In the **workflow editor**, select the workflow’s **Extract** node. Under **Output settings**, you can set **Schema-Only Output** to ON or OFF whenever you edit the workflow.

* When **Schema-Only Output** is ON, the **Extract** node returns only the JSON produced for your explicitly defined fields.

  In workflow JSON, that is the **extracted data only** layout from [Custom defined output](/concepts/structured-data-extractor/data-extractor#custom-defined-output) (no surrounding Unstructured element list).
* When **Schema-Only Output** is OFF (the default), Unstructured also emits the usual document elements and metadata alongside those extracted values.

  In workflow JSON, that is the **elements with extracted data** layout from the same [Custom defined output](/concepts/structured-data-extractor/data-extractor#custom-defined-output) section (structured fields under `DocumentData` plus the rest of the element list).

<img alt="Schema-Only Output toggle in the Extract node Output settings" />

<Note>This option is only available from the **workflow editor**.</Note>

### Additional guidance for regex

#### Regex validation

Regex validation is not the same in the browser Pipelines as at runtime: the **Start** page and **workflow editor** validate patterns with *JavaScript (ECMA-262)* before extraction runs. At runtime, including for API-defined workflows, Unstructured matches partitioned text with *Python `re`*. The subsections below describe each path.

##### Runtime

At runtime, Unstructured matches patterns by using *Python `re`*. For supported constructs, see the [Python regular expression documentation](https://docs.python.org/3/library/re.html).

##### Start page and workflow editor

On the **Start** page and in the **workflow editor**, the form validates patterns using *JavaScript (ECMA-262)* rules so you get immediate feedback before extraction runs. If you enter an invalid pattern, Pipelines prompts you to fix it and does not run extraction until the pattern is valid.

Some constructs that are valid in Python may be rejected in Pipelines, including certain conditional patterns, named groups, and named backreferences. Most common patterns are valid in both *Python `re`* and *ECMA-262*. If Pipelines rejects a pattern you need, either submit it via the API's [Extract node](/api-reference/workflow/workflows#extract-node), or simplify it to *ECMA-262*-compatible syntax. For details, see [ECMA-262 RegExp](https://tc39.es/ecma262/#sec-regexp-regular-expression-objects) or the [MDN Regular expressions guide](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions).

## Unstructured API settings for structured extraction with Regex

The following sections describe how to use the [Unstructured API](/api-reference/overview) to specify settings for
structured extraction with Regex.

### Define your schema (API only)

To specify an extraction schema with the Unstructured API, use the [Regex method of an Extract node](/api-reference/workflow/nodes/extract/extract-regex).
In this node, set the `regex_patterns` array in the `settings` object
as either as an object in a `workflow_nodes` array
(for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
[create a workflow](/api-reference/api/workflow/create-workflow),
[update a workflow](/api-reference/api/workflow/update-workflow), or
[create an on-demand workflow job](/api-reference/api/job/create-job).

### Schema-only output (API only)

You can use the `output_mode` setting with the Unstructured API to control whether
Unstructured's document elements are stripped away and returns just the extracted fields:

* Set `output_mode` to `extracted_data_only` to output only the extracted data as JSON, without any parent `DocumentData` element or any other built-in Unstructured document elements.
* Set `output_mode` to `elements_with_extracted_data` to output the extracted data as JSON, inside of a parent `DocumentData` element. This element is also
  included with any other built-in Unstructured document elements.

To specify this setting, use the [Regex method of an Extract node](/api-reference/workflow/nodes/extract/extract-regex).
In this node, set the `output_mode` key in the `settings` object.
You set this object as either as an object in a `workflow_nodes` array
(for curl) or as a `WorkflowNode` in a `WorkflowNodes` collection (for Python). This object or collection applies whenever you
[create a workflow](/api-reference/api/workflow/create-workflow),
[update a workflow](/api-reference/api/workflow/update-workflow), or
[create an on-demand workflow job](/api-reference/api/job/create-job).

### Regex pattern validation for API-defined workflows (API only)

Patterns that you send through workflow API operations are not subject to the [browser-based validation step](#regex-validation). Runtime matching still uses *Python `re`*.

<Tip>
  You can verify the list of matches for the regex `pattern` with a tool like [regex101](https://regex101.com/).

  * To test patterns for the **Start** or **workflow editor** Pipelines, set the flavor in the tool to *ECMAScript JavaScript*.
  * To test patterns for **API-defined** workflows and runtime behavior, set the flavor in the tool to *Python*.
</Tip>

### API limitations

The Unstructured API does not support the following option for structured extraction with regex. To use this option, you must use Unstructured Pipelines instead. To learn how, see the following link:

* [Visual schema builder and JSON upload/export (Pipelines only)](#define-your-schema)

## Regex output

Each of the `name` / `pattern` pairs in `regex_patterns` is extracted as an array of the matched values. The `name` field value is the name of the array in the extracted output.

See **Example settings** and **Example extracted output** below.

**Example settings:**

```json theme={null}
{
  "name": "Regex Extractor",
  "type": "structured_data_extractor",
  "subtype": "regex",
  "settings": {
    "regex_patterns": [
      {
        "name": "Price in dollars",
        "pattern": "\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?"
      }
    ],
    "output_mode": "extracted_data_only"
  }
}
```

**Note:** The `output_mode` field appears in the example settings above so the `settings` object is shown in full. It does not affect the extracted values illustrated in this example of the `Price in dollars` matches.

The extracted output includes the array `Price in dollars`.
The resulting array contains the values in the input document that match the regex `pattern`.

**Example extracted output (partial):**

```json theme={null}
{
  "Price in dollars": [
    "$249",
    "$498",
    "$189",
    "$567",
    "$59",
    "$118",
    "$1,183",
    "$224.77",
    "$1,407.77",
    "$25"
  ],
  ...
}
```

This example displays the output array for the following PDF file ([download this file](https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/pdf/invoice.pdf)).

The **example settings** above contain a JSON-escaped regex pattern.

| Context     | Pattern                                   |
| ----------- | ----------------------------------------- |
| Raw regex   | `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`      |
| JSON string | `\\$\\d{1,3}(?:,\\d{3})*(?:\\.\\d{1,2})?` |

<Tip>
  You can verify the list of matches for the regex `pattern` with a tool like [regex101](https://regex101.com/), with the flavor set to . Enter the raw regex string: `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`
</Tip>

![Sample invoice](https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/img/invoice.png)

## Regex pattern examples

The following are raw regex examples, shown without JSON escaping (as you would type them in a regex tester, not inside a JSON string value).

* Dollar amounts: `\$\d{1,3}(?:,\d{3})*(?:\.\d{1,2})?`
  * This pattern matches dollar amounts with a dollar sign immediately followed by digits (no space after `$`), comma-separated thousands, and an optional decimal part.
  * Examples that match: `$2,345.99`, `$1,000,000`
  * Examples that don't match: `$ 2,345.99` (space follows the `$`), `1,000,000` (no `$`)
* Another string format: `^[A-Z]{2}-\d{2}-\d{4}$` - This pattern matches a string with two capital letters followed by two digits and a four-digit number, separated by dashes (example: `XY-12-1234`).
  * The `^` and `$` characters in this pattern are *anchors*. Use them to require that the entire string matches the pattern.
  * Do not use anchors if you need to extract values inline in text (like `Invoice id XY-12-1234`, `Patient record AB-34-5678`, `ID: AA-00-0000`)
  * Use anchors when the whole string must match the pattern exactly, with no extra text before or after (for example, `XY-12-1234` alone).
  * Examples that match: `XY-12-1234`, `AA-00-0000`
  * Examples that don't match: `Invoice XY-12-1234`, `XY-12-1234 ` (trailing space), `xy-12-1234`

<Tip>You can test and refine patterns by using tools such as [regex101](https://regex101.com/) with the flavor set to .</Tip>


# Saving extracted data
Source: https://docs.unstructured.io/concepts/structured-data-extractor/saving-extracted-data



Unstructured recommends that you save the extracted JSON data only to a blob storage, file storage, or No-SQL database destination location.

For the extracted data only output format, Unstructured's destination connectors cannot insert custom JSON objects into a SQL-style database or vector store. Instead, save the extracted JSON to blob storage, file storage, or a NoSQL database. Then use your own code to insert the data into your target destination.

For the elements with extracted data output format, Unstructured does not recommend that you save `DocumentData` elements as rows, records, or entries within a traditional SQL-style destination database or vector store either, for the following reasons:

* Saving a mixture of `DocumentData` elements and default Unstructured elements such as `Title`, `NarrativeText`, and `Table` elements and so on in the same table, collection, or index might cause unexpected performance issues or might return less useful search and query results.
* The `DocumentData` elements' `extracted_data` contents can get quite large and complex, exceeding the column or field limits of some SQL-style databases or vector stores.

For how structured values are wrapped in workflow JSON (**elements with extracted data** vs **extracted data only**), see [Custom defined output](/concepts/structured-data-extractor/data-extractor#custom-defined-output).


# Using the structured data extractor
Source: https://docs.unstructured.io/concepts/structured-data-extractor/using-the-structured-data-extractor



<Note>
  The following information applies only to the [Unstructured Pipelines](/pipelines/overview).

  To use the Unstructured API with the structured data extractor, see the API-related sections in:

  * [Structured Extraction with LLM](/concepts/structured-data-extractor/llm-options)
  * [Structured Extraction with Regex](/concepts/structured-data-extractor/regex-options)
</Note>

The [structured data extractor](/concepts/structured-data-extractor/data-extractor) works from two places in Unstructured Pipelines:

* From the **Start** page of your Unstructured account. This approach works only with a single file that is stored on your local machine. Follow the [Start page procedure](#use-the-structured-data-extractor-from-the-start-page) below.
* From the Unstructured **workflow editor**. This approach works with a single file that is stored on your local machine, or with any number of files that are stored in remote locations. Follow the [workflow editor procedure](#use-the-structured-data-extractor-from-the-workflow-editor) below.

<h2>
  Use the structured data extractor from the Start page
</h2>

To have Unstructured [extract the data in a custom-defined format](/concepts/structured-data-extractor/data-extractor#custom-defined-output) for a single file that is stored on your local machine, do the following from the **Start** page:

1. Sign in to your Unstructured account, if you are not already signed in.

2. On the sidebar, click **Start**, if the **Start** page is not already showing.

3. In the **Welcome, get started right away!** tile, do one of the following:

   * To use a file on your local machine, click **Browse files** and then select the file, or drag and drop the file onto **Drop file to test**.

     <Note>
       If you use a local file, the file must be 50 MB or less in size.
     </Note>

   * To use a sample file provided by Unstructured, click one of the sample files that are shown, such as **realestate.pdf**.

4. After Unstructured partitions the selected file into Unstructured's document element format, click **Update results** to
   have Unstructured apply generative enrichments, such as [image descriptions](/concepts/enriching/image-descriptions) and
   [generative OCR](/concepts/enriching/generative-ocr), to those document elements.

5. In the title bar, next to **Transform**, click **Extract**.

6. In the **Define Schema** pane, choose an extraction **Method**: **LLM** (the default) or **Regex**. For a comparison of the two methods, see [Choose an extraction method: LLM or Regex](/concepts/structured-data-extractor/choose-extraction-method).

7. **LLM only:** Do one of the following to define your schema:

   <Note>All schemas must conform to the [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) guidelines. Per OpenAI's guidelines, the maximum supported JSON schema nesting depth is 10 levels.</Note>

   * To use a schema that Unstructured suggests for the selected file, click **Run Schema**.
   * To use a custom schema, click **Upload JSON**; enter your own custom schema or upload a JSON file that contains your custom schema; click **Use this Schema**; and then click **Run Schema**.
   * To use a visual editor to define the schema, click the ellipses (three dots) icon; click **Reset form**; enter your own custom schema objects and their properties; and then click **Run Schema**.
   * To use a plain language prompt, click **Suggest**; enter your prompt in the **Prompt a Schema** dialog; click **Generate schema**; make any changes as needed; and then click **Run Schema**. For more information, see [Plain language in a schema prompt](/concepts/structured-data-extractor/llm-options#prompt-a-schema).

8. **Regex only:** Do one of the following to define your schema:

   * To use the visual schema builder, for each field enter the values below. Add additional fields as needed.

     * **Pattern name** – a descriptive label for the field.
     * **Regular expression** – a regex pattern.

   * To import an existing schema, click **Upload JSON**.

   To learn more about Regex specifications, example formats, and validation tools, see [Regex-based extraction](/concepts/structured-data-extractor/regex-options).

9. The extracted data appears in the **Extract Results** pane. You can do one of the following:

   * To view a formatted view of the extracted data, click **Formatted**.
   * To view the JSON representation of the extracted data, click **JSON**.
   * To download the JSON representation of the extracted data as a local JSON file, click the download icon next to **Formatted** and **JSON**.
   * To change the schema and then re-run the extraction, click the back arrow next to **Extract Results**, and then skip back to step 7 or 8 in this procedure.

<h2>
  Use the structured data extractor from the workflow editor
</h2>

To have Unstructured [extract the data in a custom-defined format](/concepts/structured-data-extractor/data-extractor#custom-defined-output) for a single file that is stored on your local machine, or with any number of files that are stored in remote locations, do the following from the workflow editor:

1. If you already have an Unstructured workflow that you want to use, open it in the workflow editor. Otherwise, create a new
   workflow as follows:

   a. Sign in to your Unstructured account, if you are not already signed in.<br />
   b. On the sidebar, click **Workflows**.<br />
   c. Click **New Workflow +**.<br />
   d. With **Build it Myself** already selected, click **Continue**. The workflow editor appears.<br />

2. Add an **Extract** node to your Unstructured workflow. This node must be added right before the workflow's **Destination** node.
   To add this node, in the workflow designer, click the **+** (add node) button immediately before the **Destination** node, and then click **Enrich > Extract**.

3. Click the newly added **Extract** node to select it.

4. In the **structured data extractor** settings pane, under **Schema**, choose an extraction method: **LLM** (the default) or **Regex**. For a comparison of the two methods, see [Choose an extraction method: LLM or Regex](/concepts/structured-data-extractor/choose-extraction-method).

5. Under **Output settings**, switch **Schema-Only Output** on to return only the extracted fields, or off (the default) to also include Unstructured's document elements and metadata.

6. **LLM only:** For **Model**, select your provider and model. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

7. **LLM only:** To specify the custom schema, do one of the following:

   <Note>All schemas must conform to the [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) guidelines. Per OpenAI's guidelines, the maximum supported JSON schema nesting depth is 10 levels.</Note>

   * To use a custom schema, click **Upload JSON**; enter your own custom schema or upload a JSON file that contains your custom schema; and then click **Use this Schema**.
   * To use a visual editor to define the schema, enter your own custom schema objects and their properties. To clear the current schema and start over, click the ellipses (three dots) icon, and then click **Reset form**.

8. **LLM only:** The **Extraction Guidance Prompt** is available. Click **+ Add Prompt** to display the dialog, where you can add plain-language instructions that tell the LLM how to format, normalize, or summarize extracted values. When you are ready to save, click **Save Prompt**. For more information, see [Extraction guidance](/concepts/structured-data-extractor/llm-options#extraction-guidance-workflow-editor).

9. **Regex only:** To specify the custom schema, do one of the following:

   * To use the visual schema builder, for each field enter the values below. Add additional fields as needed.

     * **Pattern name** – a descriptive label for the field.
     * **Regular expression** – a regex pattern.

   * To import an existing schema, click **Upload JSON**.

   To learn more about Regex specifications, example formats, and validation tools, see [Regex-based extraction](/concepts/structured-data-extractor/regex-options).

10. To see the results of the structured data extractor, do one of the following:

* If you have already selected a local file as input to your workflow, click **Test** immediately above the **Source** node. Results appear in the **Test output** pane on-screen.
* If you are using source and destination connectors for your workflow, [run the workflow as a job](/pipelines/jobs#run-a-job),
  [monitor the job](/pipelines/jobs#monitor-a-job), and then examine the job's results in your destination location.


# Query processed PDF with HuggingChat
Source: https://docs.unstructured.io/examplecode/codesamples/api/huggingchat



This example uses the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest) or the
[Unstructured JavaScript/TypeScript SDK](/api-reference/legacy-api/partition/sdk-jsts) to send a PDF file to
the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing. Unstructured processes the PDF and extracts the PDF's content.
This example then sends some of the content to [HuggingChat](https://huggingface.co/chat/), Hugging Face's open-source AI chatbot,
along with some queries about this content.

To run this example, you'll need:

* The [hugchat](https://pypi.org/project/hugchat/) package for Python, or the [huggingface-chat](https://www.npmjs.com/package/huggingface-chat) package for JavaScript/TypeScript.
* Your Unstructured API key and API URL. [Get an API key and API URL](/api-reference/legacy-api/partition/overview).
* Your Hugging Face account's email address and account password. [Get an account](https://huggingface.co/join).
* A PDF file for Unstructured to process. This example uses a sample PDF file containing the text of the United States Constitution,
  available for download from [https://constitutioncenter.org/media/files/constitution.pdf](https://constitutioncenter.org/media/files/constitution.pdf).

These environment variables:

* `UNSTRUCTURED_API_KEY` - Your Unstructured API key value.
* `UNSTRUCTURED_API_URL` - Your Unstructured API URL.

These environment variables:

* `HUGGING_FACE_EMAIL`, representing your Hugging Face account's email address.
* `HUGGING_FACE_PASSWORD`, representing your Hugging Face account's email address.
* `HUGGING_FACE_COOKIE_DIR_PATH`, representing a login cookie cache path, typically `./cookies/` or `./login_cache/`.

This example's code is as follows:

<AccordionGroup>
  <Accordion title="Ingest Python library">
    ```python Python theme={null}
    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    import os, json
    from hugchat import hugchat
    from hugchat.login import Login
        
    def generate_json_from_local(
            input_path: str,
            output_dir: str,
            partition_by_api: bool = False,
            api_key: str = None,
            partition_endpoint: str = None,
            split_pdf_page: bool = True,
            split_pdf_allow_failed: bool = True,
            split_pdf_concurrency_level: int = 15
        ):
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=input_path),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=partition_by_api,
                api_key=api_key,
                partition_endpoint=partition_endpoint,
                additional_partition_args={
                    "split_pdf_page": split_pdf_page,
                    "split_pdf_allow_failed": split_pdf_allow_failed,
                    "split_pdf_concurrency_level": split_pdf_concurrency_level
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=output_dir)
        ).run()

    def extract_matching_texts_from_local(input_json_file_path: str, text_to_match: str) -> str:
        voting_texts = ""

        with open(input_json_file_path, 'r') as file:
            file_elements = json.load(file)

        for element in file_elements:
            if text_to_match in element["text"]:
                voting_texts += " " + element["text"]

        return voting_texts

    def log_in_to_hugging_face(email: str, passwd: str, cookie_dir_path: str) -> hugchat.ChatBot:
        sign = Login(
            email=email,
            passwd=passwd
        )

        cookies = sign.login(cookie_dir_path=cookie_dir_path)

        return hugchat.ChatBot(cookies=cookies.get_dict())

    if __name__ == "__main__":
        generate_json_from_local(
            input_path=os.getenv("LOCAL_FILE_INPUT_DIR"),
            output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
        )

        chatbot = log_in_to_hugging_face(
            email=os.getenv("HUGGING_FACE_EMAIL"),
            passwd=os.getenv("HUGGING_FACE_PASSWORD"),
            cookie_dir_path=os.getenv("HUGGING_FACE_COOKIE_DIR_PATH")
        )

        voting_texts = extract_matching_texts_from_local(
            input_json_file_path=f"{os.getenv("LOCAL_FILE_OUTPUT_DIR")}/constitution.json",
            text_to_match="vot"
        )

        print("\n-----\n")
        print("Querying HuggingChat...")
        print("\n-----\n")

        req = f"Given the following information, what is the minimum voting age in the United States? {voting_texts}"
        print(req)
        print("\n-----\n")
        print(chatbot.chat(text=req))

        print("\n-----\n")
        print("Querying HuggingChat again...")
        print("\n-----\n")

        follow_up = "And when were women given the right to vote in the United States?"
        print(follow_up)
        print("\n-----\n")

        print(chatbot.chat(text=follow_up))
    ```
  </Accordion>

  <Accordion title="JavaScript/TypeScript SDK">
    <Warning>
      Unstructured recommends that you use the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest) instead.

      The Ingest Python library provides faster processing of larger individual files, and faster and easier processing of multiple files at a time in batches.
    </Warning>

    ```typescript TypeScript theme={null}
    import { UnstructuredClient } from "unstructured-client";
    import { Strategy } from "unstructured-client/sdk/models/shared/index.js";
    import { PartitionResponse } from "unstructured-client/sdk/models/operations";

    import * as fs from "fs";
    import * as path from 'path';
    import { fileURLToPath } from 'url';
    import { Login, ChatBot} from "huggingface-chat";

    const key = process.env.UNSTRUCTURED_API_KEY || '';
    const url = process.env.UNSTRUCTURED_API_URL || '';
    const username = process.env.HUGGING_FACE_EMAIL || '';
    const userpassword = process.env.HUGGING_FACE_PASSWORD || '';
    const cookieDirPath = process.env.HUGGING_FACE_COOKIE_DIR_PATH || '';

    const inputFilepath = path.dirname(fileURLToPath(import.meta.url)) + 
                          "/local-ingest-pdf-source/constitution.pdf"

    async function getChatBot(email: string, password: string, cachePath: string): Promise<ChatBot> {
        const signin = new Login(email, password)
        const result = await signin.login(cachePath)
        const chat = new ChatBot(result)

        return chat
    }

    async function queryAndResponse(chatbot: ChatBot, query: string) {
        let data = await chatbot.chat(query)

        if (! data) {
            console.log("No data available.");
            return;
        }
        
        let reader = data.stream?.getReader();

        if (! reader) {
            console.log("No stream available.");
            return;
        }

        while (true) {
            const { done, value } = await reader?.read();
            if (done) break;
            process.stdout.write(value);
        }    
    }

    const client = new UnstructuredClient({
        security: { apiKeyAuth: key },
        serverURL: new URL(url).origin
    });

    const data = fs.readFileSync(inputFilepath);

    client.general.partition({
        partitionParameters: {
            files: {
                content: data,
                fileName: inputFilepath
            },
            strategy: Strategy.HiRes,
            splitPdfPage: true,
            splitPdfAllowFailed: true,
            splitPdfConcurrencyLevel: 15
        }
    }).then(async (res: PartitionResponse) => {
        if (res.statusCode == 200) {
            let voting_texts = ""

            if (res)
                for (const element of res) {
                    if (element.text.includes("vot")) {
                        voting_texts += " " + element.text;
                    }
            }

            let chat = await getChatBot(username, userpassword, cachePath)
            
            await chat.initialize()

            const firstQuery = 'Given the following information, what is the minimum voting age in the United States?'
            const secondQuery = 'And when were women given the right to vote in the United States?'

            console.log(`${firstQuery} ${voting_texts}\n\n-----\n`)
            await queryAndResponse(chat, firstQuery)
            console.log(`\n\n-----\n`)
            console.log(`${secondQuery}\n\n-----\n`)
            await queryAndResponse(chat, secondQuery)
        }    
    }).catch((e) => {
        if (e.statusCode) {
            console.log(e.statusCode);
            console.log(e.body);
        } else {
            console.log(e);
        }
    });
    ```
  </Accordion>
</AccordionGroup>


# Table extraction from PDF
Source: https://docs.unstructured.io/examplecode/codesamples/apioss/table-extraction-from-pdf

This section describes two methods for extracting tables from PDF files.

<Note>
  This sample code utilizes the [Unstructured Open Source](/open-source/introduction/overview "Open Source") library and also provides an alternative method the utilizing the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview).
</Note>

## Method 1: Using partition\_pdf

To extract the tables from PDF files using the [partition\_pdf](/open-source/core-functionality/partitioning#partition-pdf), set the `skip_infer_table_types` parameter to `False` and `strategy` parameter to `hi_res`.

**Usage**

```python theme={null}
from unstructured.partition.pdf import partition_pdf

fname = "example-docs/pdf/layout-parser-paper.pdf"

elements = partition_pdf(filename=fname,
                         skip_infer_table_types=False,
                         strategy='hi_res',
           )

tables = [el for el in elements if el.category == "Table"]

print(tables[0].text)
print(tables[0].metadata.text_as_html)

```

## Method 2: Using Auto Partition or Unstructured API

By default, table extraction from all file types is enabled. To extract tables from PDFs and images using [Auto Partition](/open-source/core-functionality/partitioning#partition) or [Unstructured API parameters](/api-reference/legacy-api/partition/api-parameters) simply set `strategy` parameter to `hi_res`.

**Usage: Auto Partition**

```python theme={null}
from unstructured.partition.auto import partition

filename = "example-docs/pdf/layout-parser-paper.pdf"

elements = partition(filename=filename,
                     strategy='hi_res',
           )

tables = [el for el in elements if el.category == "Table"]

print(tables[0].text)
print(tables[0].metadata.text_as_html)

```

**Usage: API Parameters**

```bash theme={null}
curl -X 'POST' \
    'https://api.unstructured.io' \
    -H 'accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -F 'files=@sample-docs/layout-parser-paper-with-table.jpg' \
    -F 'strategy=hi_res' \
    | jq -C . | less -R

```


# Multi-file API processing
Source: https://docs.unstructured.io/examplecode/codesamples/oss/multi-files-api-processing



<Note>
  This sample code utilizes the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview).
</Note>

## Introduction

This guide demonstrates how to process multiple files using the Unstructured API and S3 Connector and implement context-aware chunking. The process involves installing dependencies, configuring settings, and utilizing Python scripts to manage and chunk data effectively.

## Prerequisites

Ensure you have Unstructured API key and access to an S3 bucket containing the target files.

## Step-by-Step Process

### Step 1: Install Unstructured and S3 Dependency

Install the unstructured package with S3 support.

```bash theme={null}
pip install "unstructured-ingest[s3]"

```

### Step 2: Import Libraries

Import necessary libraries from the unstructured package for chunking and S3 processing.

```python theme={null}
from unstructured_ingest.interfaces import (
    FsspecConfig,
    PartitionConfig,
    ProcessorConfig,
    ReadConfig,
)
from unstructured_ingest.runner import S3Runner

from unstructured.chunking.title import chunk_by_title
from unstructured.staging.base import dict_to_elements

```

### Step 3: Configuration

Set up the API key and S3 URL for accessing the data.

```python theme={null}
UNSTRUCTURED_API_KEY = os.getenv('UNSTRUCTURED_API_KEY')
S3_URL = "s3://rh-financial-reports/world-development-bank-2023/"

```

### Step 4: Python Runner

Configure and run the S3Runner for processing the data.

```python theme={null}
runner = S3Runner(
     processor_config=ProcessorConfig(
         verbose=True,
         output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
         num_processes=8,
     ),
     read_config=ReadConfig(),
     partition_config=PartitionConfig(
         partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
         partition_by_api=True,
         api_key=os.getenv("UNSTRUCTURED_API_KEY"),
         strategy="hi_res",
     ),
     fsspec_config=FsspecConfig(
         remote_url=S3_URL,
     ),
 )

runner.run(anonymous=True)

```

### Step 5: Combine JSON Files from Multi-files Ingestion

Combine JSON files into a single dataset for further processing.

```json theme={null}
combined_json_data = read_and_combine_json("Connector-Output/world-development-bank-2023")

```

### Step 6: Convert into Unstructured Elements for Chunking

Convert the combined JSON data into Unstructured Elements and apply chunking by title.

```python theme={null}
elements = dict_to_elements(combined_json_data)
chunks = chunk_by_title(elements)

```

## Conclusion

Following these steps allows for efficient processing of multiple files using the Unstructured S3 Connector. The context-aware chunking helps in organizing and analyzing the data effectively.


# Delta Table source connector
Source: https://docs.unstructured.io/examplecode/codesamples/oss/table-source-connector



<Note>
  This sample code utilizes the [Unstructured Open Source](/open-source/introduction/overview "Open Source") Library.
</Note>

## Objectives

1. Extract text and metadata from a PDF file using the Unstructured.io Python SDK.

2. Process and store this data in a Databricks Delta Table.

3. Retrieve data from the Delta Table using the Unstructured.io Delta Table Connector.

## Prerequisites

* Unstructured Python SDK

* Databricks account and workspace

* AWS S3 for Delta Table storage

## Processing and Storing into Databricks Delta Table

3. Initialize PySpark

```python theme={null}
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('sparkdf').getOrCreate()

```

4. Convert JSON output into Dataframe

```python theme={null}
import pyspark

dataframe = spark.createDataFrame(res.elements)

```

5. Store DataFrame as Delta Table

```
dataframe.write.mode("overwrite").format("delta").saveAsTable("delta_table")

```

## Conclusion

This documentation covers the essential steps for converting unstructured PDF data into structured data and storing it in a Databricks Delta Table. It also outlines how to extract this data for further use.


# Vector database ingestion
Source: https://docs.unstructured.io/examplecode/codesamples/oss/vector-database



<Note> This sample code utilizes the [Unstructured Open Source](/open-source/introduction/overview "Open Source") Library. </Note>

In this guide, we demonstrate how to leverage Unstructured.IO, ChromaDB, and LangChain to summarize topics from the front page of CNN Lite. Utilizing the modern LLM stack, including Unstructured, Chroma, and LangChain, this workflow is streamlined to less than two dozen lines of code.

## Gather Links with Unstructured

First, we gather links from the CNN Lite homepage using the partition\_html function from Unstructured. When Unstructured partitions HTML pages, links are included in the metadata for each element, making link collection straightforward.

```python theme={null}
from unstructured.partition.html import partition_html

cnn_lite_url = "https://lite.cnn.com/"
elements = partition_html(url=cnn_lite_url)
links = []

for element in elements:
    if element.metadata.link_urls:
        relative_link = element.metadata.link_urls[0][1:]
        if relative_link.startswith("2024"):
            links.append(f"{cnn_lite_url}{relative_link}")

```

## Ingest Individual Articles with UnstructuredURLLoader

With the links in hand, we preprocess individual news articles using UnstructuredURLLoader. This loader fetches content from the web and then uses the unstructured partition function to extract content and metadata. Here we preprocess HTML files, but it also works with other response types like application/pdf. The result is a list of LangChain Document objects.

```python theme={null}
from langchain.document_loaders import UnstructuredURLLoader

loaders = UnstructuredURLLoader(urls=links, show_progress_bar=True)
docs = loaders.load()

```

## Load Documents into ChromaDB

The next step is to load the preprocessed documents into ChromaDB. This process involves vectorizing the documents using OpenAI embeddings and loading them into Chroma’s vector store. Once in Chroma, similarity search can be performed to retrieve documents related to specific topics.

```python theme={null}
from langchain.vectorstores.chroma import Chroma
from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings)
query_docs = vectorstore.similarity_search("Update on the coup in Niger.", k=1)

```

## Summarize the Documents

After retrieving relevant documents from Chroma, we summarize them using LangChain. The load\_summarization\_chain function allows for easy summarization, simply requiring the selection of an LLM and summarization chain.

```python theme={null}
from langchain.chat_models import ChatOpenAI
from langchain.chains.summarize import load_summarize_chain

llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo-16k")
chain = load_summarize_chain(llm, chain_type="stuff")
chain.run(query_docs)

```

## Jupyter Notebook

To delve deeper into this example, you can access the full Jupyter Notebook here: [News of the Day Notebook](https://github.com/Unstructured-IO/unstructured/blob/main/examples/chroma-news-of-the-day/news-of-the-day.ipynb)


# Notebooks
Source: https://docs.unstructured.io/examplecode/notebooks

Notebooks contain complete working sample code for end-to-end solutions.

<CardGroup>
  <Card title="Getting Started with Unstructured and Teradata" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Getting_Started_with_Unstructured_and_Teradata.ipynb">
    <br />

    Learn how to build data processing workflows using the Unstructured API to preprocess unstructured files from S3 and store the structured outputs in Teradata Vantage for retrieval.

    <br />

    `Unstructured API` `Workflows` `S3` `Teradata`

    <br />
  </Card>

  <Card title="Unstructured API: Convert Documents to Stylized HTML" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Convert_Documents_to_Stylized_HTML_using_the_Unstructured_API.ipynb">
    <br />

    Partition documents into structured elements using our VLM-based partitioning strategy, then convert the extracted HTML metadata into beautifully stylized web documents for RAG pipelines, knowledge bases, and AI-powered applications.

    <br />

    `Unstructured API` `On-demand jobs` `Local files`

    <br />
  </Card>

  <Card title="Unstructured API: On-Demand Jobs Quickstart" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_On_Demand_Jobs_Quickstart.ipynb">
    <br />

    Learn the basics of how to use the Unstructured API to run an on-demand job with a short-lived workflow that takes one or more local files as input.

    <br />

    `Unstructured API` `On-demand jobs` `Local files` `Workflow operations` `Workflow templates`

    <br />
  </Card>

  <Card title="Unstructured API: On-Demand Jobs Walkthrough" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_On_Demand_Jobs_Walkthrough.ipynb">
    <br />

    Learn how to use the Unstructured API to run an on-demand job with a short-lived workflow that takes one or more local files as input and applies basic partitioning and a few enrichments. Then, extend your knowledge of and skills
    with Unstructured by adding chunking, more enrichments, and embeddings to your on-demand jobs.

    <br />

    `Unstructured API` `On-demand jobs` `Local files` `Workflow operations` `Workflow templates`

    <br />
  </Card>

  <Card title="Adding a Memory Layer to RAG with Unstructured and Mem0" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/MemoryLayer_for_RAG.ipynb">
    <br />

    Learn how to build a personalized RAG system using Unstructured's document processing and Mem0's memory layer to create an AI assistant that remembers user preferences and adapts responses across sessions.

    <br />

    `Unstructured API` `Workflows` `RAG` `S3` `Weaviate` `Memory`

    <br />
  </Card>

  <Card title="Agentic Weekly AI News TL;DR" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Agentic-Weekly-AI-News-TLDR.ipynb">
    <br />

    Build an automated pipeline that scrapes the last 7 days of AI papers & posts (ArXiv, OpenAI, Anthropic, Hugging Face, DeepLearning.AI), processes them with Unstructured’s Hi Res partitioner for clean text, stores structured chunks in MongoDB, and generates both detailed summaries and an executive brief for a weekly newsletter.

    <br />

    `Unstructured API` `Workflows` `hi_res` `Scraping` `ArXiv` `OpenAI` `Anthropic` `Hugging Face` `DeepLearning.AI` `S3` `MongoDB` `Summarization`

    <br />
  </Card>

  <Card title="Graph RAG for Academic Papers" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/GraphRAG_for_Academic_Papers.ipynb">
    <br />

    Learn how to build a GraphRAG system for research papers using Unstructured's Named Entity Recognition to extract custom entities and relationships, then query them with Neo4j to answer complex questions that require understanding connections between models, datasets, and metrics.

    <br />

    `Unstructured API` `Workflows` `Graph RAG` `S3` `Neo4j`

    <br />
  </Card>

  <Card title="Unstructured API Walkthrough" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_Walkthrough.ipynb">
    <br />

    This walkthrough provides you with deep, hands-on experience with the Unstructured API. As you follow along, you will learn how to use many of the Unstructured API's features for partitioning, enriching, chunking, and embedding.

    <br />

    `Unstructured API` `Workflows` `Workflow Operations` `Local File`

    <br />
  </Card>

  <Card title="Everything (from) Everywhere All at Once" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/RAG_From_Multiple_Data_Sources_FileTypes.ipynb">
    <br />

    Set up an AI assistant that answers questions by querying your company's scattered data. Retrieve context from contracts in Azure, sales decks in OneDrive, and emails in Outlook through a single RAG pipeline.

    <br />

    `Unstructured API` `Workflows` `RAG` `Azure Blob Storage` `Outlook` `OneDrive` `AstraDB`

    <br />
  </Card>

  <Card title="Agentic RAG with Visually Grounded Answers and Visual Citations" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Agentic-RAG-with-Visually-Grounded-Answers-and-Visual-Citations.ipynb">
    <br />

    Learn how to build an AI-powered document processing system that extracts both text and images from PDFs in S3, generates intelligent descriptions for visual elements, and enables a searchable knowledge base that can answer questions about charts, diagrams, and product visuals.

    <br />

    `Unstructured API` `Workflows` `S3` `Image Processing` `Visual RAG` `Enterprise AI`

    <br />
  </Card>

  <Card title="Building a Hybrid RAG System: From Fragmented Data to Unified Intelligence" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_Hybrid_RAG_Pipeline_with_ElasticSearch.ipynb">
    <br />

    Learn how to build a comprehensive hybrid RAG system that processes multiple data sources simultaneously - combining S3 PDFs and Elasticsearch records into a unified knowledge base for enterprise AI applications.

    <br />

    `Unstructured API` `Workflows` `S3` `Elasticsearch` `Hybrid RAG` `Enterprise AI`

    <br />
  </Card>

  <Card title="Dropbox-to-Pinecone Connector API Quickstart" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Dropbox_To_Pinecone_Connector_Quickstart.ipynb">
    <br />

    Learn how to set up and run a custom workflow that uses a free Dropbox storage location as a source and a free Pinecone serverless index as a destination, suitable for powering RAG applications.

    <br />

    `Unstructured API` `Workflows` `Dropbox` `Pinecone`

    <br />
  </Card>

  <Card title="Getting Started with Unstructured API and PostgreSQL" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Getting_Started_with_Unstructured_API_and_PostgreSQL.ipynb">
    <br />

    Learn how to build data processing workflows using the Unstructured API and Python SDK to preprocess unstructured files from S3 and store the structured outputs in PostgreSQL for retrieval.

    <br />

    `Unstructured API` `Workflows` `S3` `PostgreSQL`

    <br />
  </Card>

  <Card title="Unstructured Partition Endpoint Quickstart" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_Partition_Endpoint_Quickstart.ipynb">
    <br />

    This notebook calls the Unstructured Python SDK to have Unstructured process a local file by using the legacy Unstructured Partition Endpoint.

    <br />

    `Unstructured API` `Partition Endpoint` `Local file`

    <br />
  </Card>

  <Card title="Preserving Table Structure for Better Retrieval" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Preserving_Table_Structure_for_Better_Retrieval.ipynb">
    <br />

    This notebook explores using Unstructured API to process financial documents while preserving tabular structure in a way that's usable by downstream applications.

    <br />

    `Unstructured API` `Workflows` `S3`  `Astra DB`

    <br />
  </Card>

  <Card title="RAG without Embeddings" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Rag_without_Embeddings.ipynb">
    <br />

    Learn how to build a RAG pipeline without any embedding models. Use Unstructured to preprocess documents, index them into Elasticsearch, and retrieve using classic BM25 scoring.

    <br />

    `Unstructured API` `Workflows` `Elasticsearch` `BM25`

    <br />
  </Card>

  <Card title="Getting Started with Unstructured API and Redis" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Getting_Started_with_Unstructured_API_and_Redis.ipynb">
    <br />

    Learn how to build data processing workflows using the Unstructured API and Python SDK to preprocess unstructured files from S3 and store the structured outputs in Redis Cloud for retrieval.

    <br />

    `Unstructured API` `Workflows` `S3` `Redis`

    <br />
  </Card>

  <Card title="Create a S3 to Qdrant Pipeline using the Unstructured API" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/S3_to_Qdrant_Workflow_using_Unstructured_API.ipynb">
    <br />

    This notebook walks through using the Unstructured API's workflow operations to set up a complete pipeline that pulls documents from S3, processes them using Unstructured, and stores the resulting embeddings in Qdrant for fast vector search and retrieval.

    <br />

    `Unstructured API` `Workflows` `S3` `Qdrant` `VLM` `Embeddings`

    <br />
  </Card>

  <Card title="Two-stage retrieval: similarity search + rerankers" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Rag_with_Reranking.ipynb">
    <br />

    Improve RAG precision with a two-stage retrieval pipeline: fast vector search followed by reranking using Cohere’s re-ranker models.

    <br />

    `Unstructured API` `Workflows` `Cohere` `Pinecone`

    <br />
  </Card>

  <Card title="Create a S3 to MongoDB Pipeline using the Unstructured API" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/S3_to_MongoDB_Workflow_using_Unstructured_API.ipynb">
    <br />

    Learn how to build an end-to-end document processing pipeline that processes PDFs from S3 and stores structured results in MongoDB. Features VLM-powered partitioning, semantic chunking, and vector embeddings using the Unstructured Workflows API.

    <br />

    `Unstructured API` `Workflows` `S3` `MongoDB` `VLM` `Embeddings`

    <br />
  </Card>

  <Card title="Getting Started with Unstructured API and IBM watsonx.data" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Azure_to_IBM_WatsonX.ipynb">
    <br />

    Learn how to create data processing workflows with Unstructured API and its Python SDK to preprocess all of your unstructured data from your Azure Blob Storage into your IBM watsonx.data instance.

    <br />

    `Unstructured API` `Workflows` `Azure Blob Storage` `IBM watsonx.data`

    <br />
  </Card>

  <Card title="Using Unstructured with Snowflake Cortex Search for RAG" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Use_Unstructured_with_Snowflake_Cortex_for_RAG_Search.ipynb">
    <br />

    Use Snowflake Cortex and RAG to do natural-language searches across a Snowflake table that contains data provided by Unstructured. Additional Snowflake Cortex functions are also explored.

    <br />

    `Unstructured API` `Snowflake Cortex` `RAG Search` `Workflows` `S3`

    <br />
  </Card>

  <Card title="Agentic RAG with LangGraph and Together AI" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/AgenticRAG_with_LangGraph,TogetherAI.ipynb">
    <br />

    Build Agentic RAG with `LangGraph` and `Together AI` and compare the results with Vanilla RAG in pure Python

    <br />

    `Unstructured API` `Workflows` `Agents` `LangGraph` `Together AI` `Astra DB`

    <br />
  </Card>

  <Card title="Getting Started with Unstructured API and Snowflake" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Getting_Started_with_Unstructured_API_and_Snowflake.ipynb">
    <br />

    Learn how to create data processing workflows with Unstructured API and its Python SDK to preprocess all of your unstructured data from your Azure Blob Storage into your Snowflake Table.

    <br />

    `Unstructured API` `Workflows` `Azure Blob Storage` `Snowflake`

    <br />
  </Card>

  <Card title="Building Graph-Based RAG Applications" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Building_Graph_Based_RAG_Applications_with_Unstructured_and_AstraDB.ipynb">
    <br />

    Learn how to use the Unstructured API to create a Graph RAG-based workflow that writes data with named entity recognition (NER) to your Astra DB.

    <br />

    `Unstructured API` `Workflows` `Graph RAG` `NER` `Astra DB`

    <br />
  </Card>

  <Card title="Getting Started with Unstructured API and Delta Tables in Databricks" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Getting_Started_with_Unstructured_API_and_Delta_Tables_in_Databricks.ipynb">
    <br />

    Learn how to create data processing workflows with Unstructured API and its Python SDK to preprocess all of your unstructured data into your Delta Table.

    <br />

    `Unstructured API` `Workflows` `Databricks` `S3`

    <br />
  </Card>

  <Card title="RAG for Online Documentation" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/RAG_for_documentation.ipynb">
    <br />

    Crawl websites with Firecrawl and build a RAG workflow powered by Unstructured and MongoDB Atlas vector search.

    <br />

    `Unstructured API` `Workflows` `MongoDB`

    <br />
  </Card>

  <Card title="Unstructured API Workflow Operations Quickstart" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_API_Workflow_Operations_Quickstart.ipynb">
    <br />

    Build an end-to-end workflow in Unstructured programmatically by using the Unstructured API's workflow operations.

    <br />

    `Unstructured API` `Workflows` `S3`

    <br />
  </Card>

  <Card title="RAG with Databricks Vector Search with Context from Multiple Sources" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Delta_Tables_Databricks_Multiple_Sources.ipynb">
    <br />

    Build RAG with Databricks Vector Search with context preprocessed from multiple sources by Unstructured.

    <br />

    `Databricks` `Introductory notebook`

    <br />
  </Card>

  <Card title="Agentic RAG with Hugging Face smolagents vs Vanilla RAG" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Agentic_RAG_with_HuggingFace_smolagents.ipynb">
    <br />

    Build Agentic RAG with `smolagents` library and compare the results with Vanilla RAG in pure Python

    <br />

    `GPT-4o` `smolagents` `Agents` `DataStax` `S3` `Advanced notebook`

    <br />
  </Card>

  <Card title="LLama3.2 RAG evaluation on unstructured text" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Llama3_2_RAG_evaluation_on_Unstructured_Text_via_VLM.ipynb">
    <br />

    Evaluate Llama3.2 for your RAG system with Unstructured, GPT-4o, Ragas, and LangChain

    <br />

    `GPT-4o` `Ragas` `LangChain` `Llama3.2` `Pinecone` `S3` `Advanced notebook`

    <br />
  </Card>

  <Card title="Multimodal RAG: Enhancing RAG outputs with image results" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Multimodal_RAG_with_image_results.ipynb">
    <br />

    Process a file in S3 with Unstructured and return images in your RAG output

    <br />

    `S3` `FAISS` `GPT-4o-mini` `Advanced notebook`

    <br />
  </Card>

  <Card title="Quantitative Reasoning with tables inside PDFs" href="https://app.hex.tech/5e6b6e24-dead-4d3b-b9da-a9a7ad587b96/hex/cb595d8a-6eac-4e19-96ed-b1540e5c031c/draft/logic">
    <br />

    From Pixels to Insights: Seamlessly Extracting and Visualizing Table Data with Unstructured and Hex

    <br />

    `Unstructured API` `Hex` `Advanced notebook`

    <br />
  </Card>

  <Card title="PII removal with GLiNER in unstructured data ETL" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/PII_removal_in_unstructured_data_ETL.ipynb">
    <br />

    Remove Personally Identifiable Information (PII) as a part of unstructured data preprocessing.

    <br />

    `Unstructured API`  `PII` `GLiNER` `Advanced notebook`

    <br />
  </Card>

  <Card title="Custom metadata extraction and self-querying retrieval" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/custom_metadata_self_querying_rag_mongodb_unstructured_langgraph.ipynb">
    <br />

    Extract custom metadata, and enable metadata pre-filtering in your RAG.

    <br />

    `Unstructured API` `MongoDB`  `Metadata` `Advanced notebook`

    <br />
  </Card>

  <Card title="Selecting an embedding model for custom data" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Selecting_an_embedding_model_for_custom_data.ipynb">
    <br />

    End-to-end data processing pipeline using Unstructured Serverless API.

    <br />

    `Unstructured API` `Hugging Face` `Advanced notebook`

    <br />
  </Card>

  <Card title="RAG with PDFs, LangChain and Llama 3" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/RAG_Llama3_Unstructured_LangChain.ipynb">
    <br />

    A RAG system with the Llama 3 model from Hugging Face.

    <br />

    `Unstructured API`  `🤗 Hugging Face` `LangChain` `Llama 3` `Introductory notebook`
  </Card>

  <Card title="Unstructured data ETL from S3 to SingleStore DB" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_data_ETL_from_S3_to_SingleStore.ipynb">
    <br />

    Learn to ingest, partition, chunk, embed and load data from an S3 bucket into SingleStore DB.

    <br />

    `Unstructured API`  `SingleStoreDB` `AWS S3` `Introductory notebook`
  </Card>

  <Card title="Google Drive to DataStax Astra DB" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_Google_Docs_to_Astra.ipynb">
    <br />

    Embed your Google Drive Docs in an Astra Vector Database with Unstructured Serverless API

    <br />

    `Unstructured API` `Google` `DataStax` `Introductory notebook`

    <br />
  </Card>

  <Card title="Weaviate RAG quickstart" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_Weaviate_Quickstart_OpenAI.ipynb">
    <br />

    Embed your local documents in an Weaviate Vector Database with Unstructured Serverless API

    <br />

    `Unstructured API` `OpenAI` `Weaviate` `Introductory notebook`

    <br />
  </Card>

  <Card title="Preprocess PDFs in AWS S3, load into Elasticsearch" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/S3_to_Elasticsearch_with_Unstructured.ipynb">
    <br />

    Ingest PDF documents from an S3 bucket, transform them into a normalized JSON with Unstructured Serverless API, chunk, embed and load into Elasticsearch.

    <br />

    `Unstructured API` `AWS S3` `Elasticsearch` `Introductory notebook`

    <br />
  </Card>

  <Card title="Preprocess documents in Google Drive, load into Databricks Volume" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/GoogleDrive_to_Databricks_Connector.ipynb">
    <br />

    Preprocess documents from a Google Drive Unstructured Serverless API and load them into Databricks Volume.

    <br />

    `Unstructured API` `Google Drive` `Databricks` `Introductory notebook`

    <br />
  </Card>

  <Card title="Source references in RAG responses" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/RAG_on_arXiv_papers_with_source_references.ipynb">
    <br />

    Add document source references to RAG responses based on documents metadata.

    <br />

    `Unstructured API` `RAG` `LangChain` `Intermediate notebook`

    <br />
  </Card>

  <Card title="Query processed PDF with HuggingChat" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/PDF_with_Unstructured_and_HuggingChat.ipynb">
    <br />

    Send a PDF to Unstructured for processing, and send a subset of the returned PDF's processed text to [HuggingChat](https://huggingface.co/chat/) for chatbot-style querying.

    <br />

    `Unstructured API`  `🤗 Hugging Face` `🤗 HuggingChat` `Introductory notebook`
  </Card>

  <Card title="Llama 3 Local RAG with emails" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Local_RAG_with_emails.ipynb">
    <br />

    Build a local RAG app for your emails with Unstructured, LangChain and Ollama.

    <br />

    `Unstructured API` `LangChain` `Ollama` `Llama 3` `Introductory notebook`
  </Card>

  <Card title="Building RAG With PowerPoint presentations" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Building_RAG_with_Powerpoint_presentations.ipynb">
    <br />

    A RAG solution that is based on PowerPoint files.

    <br />

    `Unstructured API`  `🤗 Hugging Face` `LangChain` `Llama 3` `Introductory notebook`
  </Card>

  <Card title="Synthetic test dataset generation" href="https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/RAG_synthetic_test_data_with_Unstructured_GPT_4o_and_Ragas.ipynb">
    <br />

    Build a Synthetic Test Dataset for your RAG system in 5 easy steps

    <br />

    `Unstructured API` `GPT-4o` `Ragas` `LangChain` `Advanced notebook`

    <br />
  </Card>
</CardGroup>


# Azure Blob Storage event triggers
Source: https://docs.unstructured.io/examplecode/tools/azure-storage-events



You can use Azure Blob Storage events, such as adding new files to—or updating existing files within—Azure Blob Storage containers, to automatically run Unstructured ETL+ workflows
that rely on those containers as sources. This enables a no-touch approach to having Unstructured automatically process new and updated files in Azure Blob Storage containers as they are added or updated.

This example shows how to automate this process by adding a custom [Azure Function](https://learn.microsoft.com/azure/azure-functions/functions-overview) app to your Azure account. This function app runs
a function whenever a new or updated file is detected in the specified Azure Blob Storage container. This function then calls the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow within your Unstructured account.

<Note>
  This example uses a custom Azure function that you create and maintain.
  Any issues with file detection, timing, or function invocation could be related to your custom function,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  function's invocation traces first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

## Step 1: Create an Azure Function App

1. Sign in to your [Azure portal](https://portal.azure.com).

2. Click **+ Create a resource**.

   If **Function App** is not visible, in **Search services and marketplace** field, enter **Function App**.

3. Next to **Function App**, click **Create** or **Create > Function App**.

4. Under **Select a hosting option**, select the radio button next to **Consumption** to create an app that is most compatible with JavaScript.

5. Click **Select**.

6. On the **Basics** tab, set the following function app settings:

   | Setting               | Suggested value           | Description                                                                                                                                                                                                                                                                                                                                                                                                                  |
   | --------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Subscription**      | Your subscription         | The Azure subscription within which to create your new function app.                                                                                                                                                                                                                                                                                                                                                         |
   | **Resource Group**    | **Create new**            | After you click **Create new**, enter some name for the new resource group within which to create your new function app. You should create a new resource group because there are known limitations when creating new function apps in an existing resource group. [Learn more](https://learn.microsoft.com/azure/azure-functions/functions-scale#limitations-for-creating-new-function-apps-in-an-existing-resource-group). |
   | **Function App name** | Some globally unique name | Some name that identifies your new function app. Valid characters are `a`-`z` (case insensitive), `0`-`9`, and `-`.                                                                                                                                                                                                                                                                                                          |
   | **Operating System**  | **Windows**               | Choose the operating system for your function app. This example uses Windows.                                                                                                                                                                                                                                                                                                                                                |
   | **Runtime stack**     | **Node.js**               | Choose a runtime that supports your favorite function programming language. This example uses JavaScript (Node.js).                                                                                                                                                                                                                                                                                                          |
   | **Version**           | **20 LTS**                | Choose the version of your selected runtime. This example uses Node.js 20 LTS.                                                                                                                                                                                                                                                                                                                                               |
   | **Region**            | Your preferred region     | Select a region that's near you or near other services that your function can access.                                                                                                                                                                                                                                                                                                                                        |

7. Click **Review + create**.

8. Click **Create**, and wait for the deployment to complete.

9. After the deployment is complete, click **Go to resource**.

## Step 2: Create a function

1. With the function app open from the previous step, on the sidebar, click **Overview**.

2. On the **Functions** tab, under **Create in Azure portal**, click **Create function**.

3. For **Select a template**, select **Azure Blob Storage trigger**, and then click **Next**.

4. For **Template details**, review the following values:

   | Setting                        | Suggested value            | Description                                                                                                |
   | ------------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------------------- |
   | **Function name**              | `BlobTrigger1`             | The name of the function to create. You can leave the default function name.                               |
   | **Path**                       | `samples-workitems/{name}` | The path to the Azure Blob Storage account that the function will monitor. You can leave the default path. |
   | **Storage account connection** | `AzureWebJobsStorage`      | You can leave the default storage account connection name.                                                 |

   <img alt="Azure Function template details" />

5. Click **Create**. The function is created, and the **Code + Test** page appears.

## Step 3: Customize the function for your workflow

1. With the **Code + Test** page open from the previous step, on the **Code + Test** tab, replace the content of the `index.js` file with the following code:

   ```javascript theme={null}
   module.exports = async function (context, myBlob) {
       context.log("JavaScript blob trigger function processed blob \n Blob:", context.bindingData.blobTrigger, "\n Blob Size:", myBlob.length, "Bytes");

       const apiKey = process.env.UNSTRUCTURED_API_KEY;
       const apiUrl = process.env.UNSTRUCTURED_API_URL;
       const headers = {
           "accept": "application/json",
           "unstructured-api-key": apiKey
       };

       try {
           const response = await fetch(apiUrl, {
               method: "POST",
               headers: headers
           });

           const data = await response.json();
           context.log("POST response:", data);
       } catch (error) {
           context.log.error("Error calling external API:", error);
       }
   };
   ```

2. Click **Save**.

3. In the navigation breadcrumb toward the top of the page, click your function app's name. The function app's settings page appears.

4. In the sidebar, expand **Settings**, and then click **Environment variables**.

5. Click **+ Add**.

6. For **Name**, enter `UNSTRUCTURED_API_URL`.

7. For **Value**, enter your `<unstructured-api-url>/workflows/<workflow-id>/run`, and replace the following placeholders:

   * Replace `<unstructured-api-url>` with your Unstructured API's workflow operations value.
   * Replace `<workflow-id>` with the ID of your Unstructured workflow. For now, because the workflow does not yet exist, enter some fictitious value, such as `1234567890`. You will
     update this value later in Step 6 after you create the workflow.

   The **Value** should now look similar to the following:

   ```text theme={null}
   https://platform.unstructuredapp.io/api/v1/workflows/1234567890/run
   ```

8. Click **Apply**.

9. Click **+ Add** again.

10. For **Name**, enter `UNSTRUCTURED_API_KEY`.

11. For **Value**, enter your Unstructured API key value.

12. Click **Apply**.

13. Click **Apply** again, and then click **Confirm**.

## Step 4: Create the Azure Storage container

1. With the function app's settings page open from the previous step, in the sidebar, click **Overview**.
2. Expand **Essentials**.
3. Next to **Resource group**, click the resource group link. The resource group's settings page appears.
4. In the sidebar, click **Overview**.
5. On the **Resources** tab, click the link next to **Storage account**. The storage account's settings page appears.
6. In the sidebar, click **Overview**.
7. On the **Properties** tab, click **Blob service**.
8. Click **Add container**.
9. For **Name**, enter `samples-workitems`. This name must match the container name that you specified earlier in the **Path** field (not including `/{name}`) in Step 2.
10. Click **Create**.

## Step 5: Create the Unstructured workflow

1. Create an Azure Blob Storage source connector in your Unstructured account. [Learn how](/pipelines/sources/azure-blob-storage). This source connector must reference
   the Azure Storage container that you created earlier in Step 4.
2. Create a new—or identify an existing—[destination connector](/pipelines/destinations/overview) in your Unstructured account.
3. Create a workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

## Step 6: Add the workflow's ID to the function's environment variables

1. Note the ID of the workflow that you created earlier in Step 5.
2. In the Azure portal, with the storage account's settings page open from Step 4, in the navigation breadcrumb toward the top of the page, click your resource group's name. The resource group's settings page appears.
3. On the **Resources** tab, click the link next to **Function App**. The function app's settings page appears.
4. In the sidebar, expand **Settings**, and then click **Environment variables**.
5. Click `UNSTRUCTURED_API_URL`.
6. Click the eyball (**Reveal password**) icon.
7. Replace the fictitious workflow ID from earlier in Step 3 with the ID of the workflow that you created earlier in Step 5.
8. Click **Apply**.
9. Click **Apply** again, and then click **Confirm**.

## Step 7: Trigger the function

1. With the function app’s settings page open from the previous step, in the navigation breadcrumb toward the top of the page, click your resource group's name. The resource group's settings page appears.
2. On the **Resources** tab, click the link next to **Storage account**. The storage account's settings page appears.
3. In the sidebar, click **Overview**.
4. On the **Properties** tab, click **Blob service**.
5. Click the **samples-workitems** link.
6. Click **Upload**, and follow the on-screen instructions to upload a file to the container.

   <Note>
     If you are unable to upload a file to the container, click **Access Control (IAM)** in the sidebar and add an appropriate role assignment that
     enables uploading files to the container, such as **Storage Blob Data Owner**. [Learn how](https://learn.microsoft.com/azure/storage/blobs/assign-azure-role-data-access?tabs=portal)
   </Note>

## Step 8: View trigger results

1. In Unstructured Pipelines for your account, click **Jobs** on the sidebar.
2. In the list of jobs, click the newly running job for your workflow.
3. After the job status shows **Finished**, go to your destination location to see the results.

## Step 9 (Optional): Delete the Azure resource group

If you are done with this example and do not want to keep the resource group in your account, you can permanently delete it as follows:

1. In the Azure portal, with the storage account's settings page open from Step 7, in the navigation breadcrumb toward the top of the page, click your resource group's name. The resource group's settings page appears.
2. Click **Delete resource group**.
3. Enter the resource group's name, and then click **Delete**. The resource group, along with the function app, storage account, and other related resources, are permanently deleted.


# Claude Desktop
Source: https://docs.unstructured.io/examplecode/tools/claude



[Claude](https://claude.com/product/overview) is an AI platform built by Anthropic. Claude excels at AI-driven tasks involving language, reasoning, analysis, coding, and more. Claude Desktop gives you a graphical interface built for running multiple Claude sessions side by side. Claude Desktop provides access to three modes of working:

* [Claude Chat](https://claude.com/product/overview), which provides general conversation with no file access, similar to [claude.ai](https://claude.ai).
  At the time this article was initially published, Claude Chat was available on all [Claude pricing plans](https://claude.com/pricing).
* [Claude Cowork](https://claude.com/product/cowork), an autonomous background agent that works on tasks in a cloud virtual machine
  with its own environment. It can run independently while you do other work. At the time this article was initially published,
  Claude Cowork was available only on Claude Pro, Max, Team, and Enterprise [pricing plans](https://claude.com/pricing).
* [Claude Code](https://claude.com/product/claude-code), an interactive coding assistant with direct access to your local files. You
  review and approve each change in real time. At the time this article was initially published, Claude Code was available only on Claude Pro, Max, Team, and Enterprise [pricing plans](https://claude.com/pricing).

This article shows how you can use the Claude Cowork interface within Claude Desktop to interact with Unstructured
[document elements and metadata](/concepts/document-elements) that have been sent to various [destinations](/pipelines/connectors#destinations) that Unstructured supports. For example, you could send through Unstructured your organization's source documents stored for instance in an Amazon S3 bucket that you own, which transforms those documents' contents into AI-ready data. You could then have Unstructured send its outputs for instance into a DataStax Astra database that you own. You could then connect Claude Desktop to this database and, finally, use Claude Cowork to interact with the data in this database.

## Requirements

Before you can use the Claude Cowork interface within Claude Desktop to interact with your destination location that
contains Unstructured outputs, you must do the following:

1. [Sign up for a Claude paid plan](https://claude.ai/login) or [upgrade from a free to a paid Claude plan](https://claude.ai/upgrade),
   if you have not already done so.

2. [Install Claude Desktop](https://support.claude.com/en/articles/10065433-install-claude-desktop) on your local machine, if you
   have not already done so.

3. [Create an Unstructured workflow](/pipelines/workflows#create-a-workflow) that connects one or more supported [sources](/pipelines/sources/overview)
   containing your organization's source documents, semi-structured data, or both, to a supported [destination](/pipelines/destinations/overview) that will contain Unstructured's AI-ready outputs based on those source documents, semi-structured data, or both. This article has been tested with the following Unstructured destination types:

   * Astra DB
   * Google Drive
   * S3

   This list will be updated over time as additional Unstructured destination types are tested.

   Any supported Unstructured destination type that has a corresponding available
   [Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/learn/server-concepts) server should work.
   Most of these MCP servers are available through third parties and not through Unstructured.

   <Info>
     MCP servers that are offered by third parties are neither owned nor supported by Unstructured.
     For questions, issues, or feature requests for a specific MCP server, contact that MCP server's provider.
   </Info>

4. Enhance your workflow output's relevance, context, and insights by adding [enrichments](/concepts/enriching/overview) as desired.

5. Add [chunking](/concepts/chunking) and [embedding](/concepts/embedding) to your workflow as desired.

   <Tip>
     While Claude Cowork can handle large files and extensive context, you do not need to manually chunk data first, but doing so can lead to better performance and efficiency for very large tasks.

     For very large-scale use cases involving millions of Unstructured outputs, embeddings are recommended for Claude Cowork, but they are not required for standard, file-based tasks.
   </Tip>

6. [Run your workflow as a job](/pipelines/jobs#run-a-job). This makes Unstructured generate AI-ready outputs based on your source documents, semi-structured data,
   or both, and store its outputs in your destination.

7. [Monitor your workflow's job run](/pipelines/jobs#monitor-a-job). After your job has successfully completed, continue by finding the following section that
   matches your Unstructured destination type and completing that section's instructions, to connect Claude Desktop to your destination.

This article has been tested with the following Unstructured destination types:

* [Astra DB](#astra-db)
* [Google Drive](#google-drive)
* [S3](#s3)

This list will be updated over time as additional Unstructured destination types are tested.

## Astra DB

<Info>
  This section references a third-party MCP server that is neither owned nor supported by Unstructured.
  For technical support, questions, issues, or feature requests for this MCP server, contact the MCP server's provider.

  For more information, see the [Astra DB MCP Server](https://github.com/datastax/astra-db-mcp) repository in GitHub.
</Info>

After you have completed the [requirements](#requirements), do the following:

1. In the [README](https://github.com/datastax/astra-db-mcp/blob/main/README.md) file in the
   [Astra DB MCP Server](https://github.com/datastax/astra-db-mcp) repository in GitHub, follow the instructions in these sections, in this order:

   1. [Prerequisites](https://github.com/datastax/astra-db-mcp/blob/main/README.md#prerequisites)
   2. [Claude Desktop](https://github.com/datastax/astra-db-mcp/blob/main/README.md#claude-desktop)

2. Start Claude Desktop. If it is already running, stop and then restart it. This step helps to ensure that Claude Desktop can successfully
   start and use the MCP server.

3. Sign in to Claude by following Claude Desktop's on-screen instructions.

4. Switch to Claude Cowork mode: in the upper-left corner of Claude Desktop, click the **Cowork** icon.

5. In the **How can I help you today** box, enter your question about the Unstructured outputs within your target Astra DB collection,
   and then press `Enter`. Repeat this step as needed.

   <Tip>
     To constrain Claude Cowork to use only this MCP server, you should be explicit
     in your sessions' initial prompt or within your Claude project's files such as `CLAUDE.md`, for example:

     ```text theme={null}
     Use ONLY the Astra DB collection "<your-collection-name>" as your source of truth.

     Rules:

     - For every question, retrieve relevant records using the
       mcp__astra-db-mcp__VectorSearch or HybridSearch tool against that
       collection before answering.
     - Do NOT use web search, WebFetch, your own training knowledge, or
       any other MCP connector. Do NOT guess or fill in gaps from memory.
     - Base every factual claim on retrieved records. Quote or paraphrase
       the retrieved `content` field and cite the source `filename` and
       `page_number` from each record's metadata.
     - If the collection does not contain enough information to answer,
       say exactly: "The collection does not contain information on this."
       Do not speculate or supplement from other sources.
     - If I ask something that requires tools other than Astra DB (such as 
       file creation or calculations on retrieved data), you may use those
       tools, but the underlying facts must still come from the collection.
     ```
   </Tip>

## Google Drive

<Info>
  This section references a third-party MCP server that is neither owned nor supported by Unstructured.
  For technical support, questions, issues, or feature requests for this MCP server, contact the MCP server's provider.

  For more information, see the [Google Drive MCP Server](https://github.com/piotr-agier/google-drive-mcp) repository in GitHub.
</Info>

After you have completed the [requirements](#requirements), do the following:

1. In the [README](https://github.com/piotr-agier/google-drive-mcp/blob/master/README.md) file in the
   [Google Drive MCP Server](https://github.com/piotr-agier/google-drive-mcp) repository in GitHub, follow the instructions in these sections, in this order:

   1. [Requirements](https://github.com/piotr-agier/google-drive-mcp/blob/master/README.md#requirements)
   2. [Google Cloud Setup](https://github.com/piotr-agier/google-drive-mcp/blob/master/README.md#google-cloud-setup)
   3. [Configuration](https://github.com/piotr-agier/google-drive-mcp/blob/master/README.md#configuration)
   4. [Usage with Claude Desktop](https://github.com/piotr-agier/google-drive-mcp/blob/master/README.md#usage-with-claude-desktop)

2. Start Claude Desktop. If it is already running, stop and then restart it. This step helps to ensure that Claude Desktop can successfully
   start and use the MCP server.

3. Sign in to Claude by following Claude Desktop's on-screen instructions.

4. Switch to Claude Cowork mode: in the upper-left corner of Claude Desktop, click the **Cowork** icon.

5. In the **How can I help you today** box, enter your question about the Unstructured outputs within your Google Drive, and then press `Enter`.
   Repeat this step as needed.

   <Tip>
     To constrain Claude Cowork to use only this MCP server, you should be explicit
     in your sessions' initial prompt or within your Claude project's files such as `CLAUDE.md`, for example:

     ```text theme={null}
     For this entire task, use only the `google-drive` MCP server to answer my questions.

     Rules:

     - Do NOT use web search, WebFetch, your own training knowledge, or
       any other MCP connector. Do NOT guess or fill in gaps from memory.
     - Base every factual claim on retrieved records. Quote or paraphrase
       the retrieved content and cite the source `filename` and
       `page_number` from each record's metadata.
     - If the files do not contain enough information to answer,
       say exactly: "The source files do not contain information on this."
       Do not speculate or supplement from other sources.
     - If I ask something that requires tools other than `google-drive`, you may use those
       tools, but the underlying facts must still come from the source files.
     ```
   </Tip>

## S3

<Info>
  This section references a third-party MCP server that is neither owned nor supported by Unstructured.
  For technical support, questions, issues, or feature requests for this MCP server, contact the MCP server's provider.

  For more information, see the [S3 MCP Server](https://github.com/samuraikun/aws-s3-mcp) repository in GitHub.
</Info>

After you have completed the [requirements](#requirements), do the following:

1. Start Claude Desktop.

2. Open your Claude Desktop config file: in the lower-left corner of Claude Desktop, click your login icon, and then click **Settings**.

3. In the sidebar, under **Desktop app**, click **Developer**.

4. Click **Edit Config**.

5. Follow the on-screen instructions to open your `claude_desktop_config.json` file in your favorite text or code editor.

6. In the file's `mcpServers` section, add the following code. (The ellipses indicate omitted code for brevity.) Replace the following placeholders,
   and then save and close the file:

   * Replace `<aws-region>` with the short ID of the AWS Region for the target S3 buckets (for example, `us-east-1`).

   * Replace `<s3-bucket>` with the name of each target S3 bucket, one entry per target bucket, with multiple bucket names
     separated by commas.

     <Note>
       Specify only bucket names here. Do not specify paths within buckets. To restrict Claude to use specific paths within buckets,
       specify those paths as a constraint within your prompts or Claude project files.
     </Note>

   * Replace `<s3-max-buckets>` with the number of specified buckets.

   * Replace `<your-aws-access-key-id>` with an AWS access key ID that has access to the buckets' contents.

   * Replace `<your-aws-secret-access-key>` with the AWS secret access key for the preceding AWS access key ID.

   ```json theme={null}
   {
     "mcpServers": {
       "...": "...",
       "aws-s3-mcp": {
         "command": "npx",
         "args": ["aws-s3-mcp", "--stdio"],
         "env": {
           "AWS_REGION": "<aws-region>",
           "S3_BUCKETS": "<s3-bucket-1>,<s3-bucket-2>,<s3-bucket-N>",
           "S3_MAX_BUCKETS": "<s3-max-buckets>",
           "AWS_ACCESS_KEY_ID": "<your-aws-access-key-id>",
           "AWS_SECRET_ACCESS_KEY": "<your-aws-secret-access-key>"
         }
       },
       "...": "..."
     },
     "...": "..."
   }
   ```

7. Restart Claude Desktop. This step helps to ensure that Claude Desktop can successfully start and use the MCP server.

8. Sign in to Claude by following Claude Desktop's on-screen instructions.

9. Switch to Claude Cowork mode: in the upper-left corner of Claude Desktop, click the **Cowork** icon.

10. In the **How can I help you today** box, enter your question about the Unstructured outputs within your target S3 bucket,
    and then press `Enter`. Repeat this step as needed.

    <Tip>
      To constrain Claude Cowork to use only this MCP server, you should be explicit
      in your sessions' initial prompt or within your Claude project's files such as `CLAUDE.md`, for example:

      ```text theme={null}
      For this entire task, use only the `aws-s3-mcp` MCP server to answer my questions. 

      Rules:

      - Do NOT use web search, WebFetch, your own training knowledge, or
        any other MCP connector. Do NOT guess or fill in gaps from memory.
      - Base every factual claim on retrieved records. Quote or paraphrase
        the retrieved content and cite the source `filename` and
        `page_number` from each record's metadata.
      - If the files do not contain enough information to answer,
        say exactly: "The source files do not contain information on this."
        Do not speculate or supplement from other sources.
      - If I ask something that requires tools other than `aws-s3-mcp`, you may use those
        tools, but the underlying facts must still come from the source files.
      ```
    </Tip>


# CrewAI
Source: https://docs.unstructured.io/examplecode/tools/crewai



[CrewAI](https://www.crewai.com/) is a popular framework for building AI agents and multi-agent workflows.

This article provides a hands-on, step-by-step walkthrough that uses [CrewAI open source](https://www.crewai.com/open-source),
along with the [Unstructured API's workflow operations MCP Server](/examplecode/tools/mcp) and Python, to
build a multi-agent workflow. This multi-agent workflow uses the MCP server to call various functions within the
[Unstructured API's workflow operations](/api-reference/workflow/overview) to build Unstructured ETL+ workflows from connector creation all
the way through to workflow job completion. This walkthrough uses an Amazon S3 bucket as both the workflow's source and destination. However, you can modify this multi-agent workflow
later to use a different S3 bucket or even different sources and destinations, to have a collection of AI agents quickly build multiple Unstructured ETL+ workflows
on your behalf with varying configurations.

## Requirements

To complete this walkthrough, you must first have:

* An Unstructured account and an Unstructured API key for that account.
* An Anthropic account and an Anthropic API key for that account.
* A properly configured Amazon S3 bucket with bucket access credentials.
* A Firecrawl account and a Firecrawl API key for that account.
* Python and the CrewAI open-source toolchain installed on your local development machine.

The following sections describe how to get these requirements.

### Unstructured account and API key

Before you begin, you must have an Unstructured account and an Unstructured API key, as follows:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

### Anthropic account and API key

This walkthrough uses a Claude Opus 3 model from Anthropic. So, before you begin, you must also have an Anthropic account and an Anthropic API key for that account.
[Sign in to or create your Anthropic account](https://console.anthropic.com/login). After you sign in to your
Anthropic account, [get your Anthropic API key](https://console.anthropic.com/settings/keys).

### Amazon S3 bucket and access credentials

This walkthrough uses an Amazon S3 bucket as both the workflow's source and destination. So, before you begin, you must also have a properly configured Amazon S3 bucket, with bucket access credentials
consisting of an access key ID and a secret access key for the AWS IAM user that has access to the bucket.
Follow the S3 connector instructions to
[create and configure the bucket and get the bucket access credentials](/pipelines/sources/s3) if you do not already have this all set up. (In these
instructions, do not follow the directions to use Unstructured Pipelines to create the S3 source connector. CrewAI will do this for you
later automatically.)

This walkthrough expects two folders to exist within the bucket, as follows:

* An `input` folder, which contains the files to process. This `input` folder must contain at least one file to process.
  If you do not have any files available to upload into the `input` folder, you can get some from the [example-docs](https://github.com/Unstructured-IO/unstructured/tree/main/example-docs) folder in the Unstructured open source library repository on GitHub.
* An `output` folder, which will contain the processed files' data from Unstructured after CrewAI runs the workflow.
  This `output` folder should be empty for now.

### Firecrawl account and API key

This walkthrough uses Firecrawl to monitor job statuses. So, before you begin, you must also have a Firecrawl account and a Firecrawl API key for that account.
[Sign in to or create your Firecrawl account](https://www.firecrawl.dev/signin/signup). After you sign in to your Firecrawl account, [get your Firecrawl API key](https://www.firecrawl.dev/app/api-keys).

### Python and CrewAI open-source toolchain and project setup

Before you can start coding on your local machine, you must install Python, and you should also install a Python package and project manager to manage your project's code dependencies.
This walkthrough uses the popular Python package and project manager [uv](https://docs.astral.sh/uv/) (although `uv` is not required to use CrewAI or the Unstructured API's workflow operations MCP Server).

<Steps>
  <Step title="Install uv">
    To install `uv`, run one of the following commands, depending on your operating system:

    <Tabs>
      <Tab title="macOS, Linux">
        To use `curl` with `sh`:

        ```bash theme={null}
        curl -fsSL https://get.uv.dev | bash
        ```

        To use `wget` with `sh` instead:

        ```bash theme={null}
        wget -qO- https://astral.sh/uv/install.sh | sh
        ```
      </Tab>

      <Tab title="Windows">
        To use PowerShell with `irm` to download the script and run it with `iex`:

        ```powershell theme={null}
        powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
        ```
      </Tab>
    </Tabs>

    If you need to install `uv` by using other approaches such as PyPI, Homebrew, or WinGet,
    see [Installing uv](https://docs.astral.sh/uv/getting-started/installation/).
  </Step>

  <Step title="Install CrewAI open source">
    Use `uv` to install the CrewAI open-source toolchain, by running the following command:

    ```bash theme={null}
    uv tool install crewai
    ```
  </Step>

  <Step title="Install Python">
    <Note>
      CrewAI open source works only with Python 3.10, 3.11, and 3.12.
    </Note>

    `uv` will detect and use Python if you already have it installed.
    To view a list of installed Python versions, run the following command:

    ```bash theme={null}
    uv python list
    ```

    If, however, you do not already have Python installed, you can install a version of Python for use with `uv`
    by running the following command. For example, this command installs Python 3.12 for use with `uv`:

    ```bash theme={null}
    uv python install 3.12
    ```
  </Step>
</Steps>

## Build and run the CrewAIproject

You are now ready to start coding.

<Steps>
  <Step title="Create the project directory">
    Switch to the directory on your local development machine where you want to create the project directory for this walkthrough.
    This example creates a project directory named `crewai_unstructured_demo` within your current working directory and then
    switches to this new project directory:

    ```bash theme={null}
    mkdir crewai_unstructured_demo
    cd crewai_unstructured_demo
    ```
  </Step>

  <Step title="Intiialize the project">
    From within the new project directory, use `uv` to initialize the project by running the following command:

    ```bash theme={null}
    uv init
    ```
  </Step>

  <Step title="Create a venv virtual environment">
    To isolate and manage your project's code dependencies, you should create a virtual environment. This walkthrough uses
    the popular Python virtual environment manager [venv](https://docs.python.org/3/library/venv.html) (although `venv` is not required to use CrewAI or the Unstructured API's workflow operations MCP Server).
    From the root of your project directory, use `uv` to create a virtual environment with `venv` by running the following command:

    ```bash theme={null}
    # Create the virtual environment by using the current Python version:
    uv venv

    # Or, if you want to use a specific Python version:
    uv venv --python 3.12
    ```
  </Step>

  <Step title="Activate the virtual environment">
    To activate the `venv` virtual environment, run one of the following commands from the root of your project directory:

    <Tabs>
      <Tab title="macOS, Linux">
        * For `bash` or `zsh`, run `source .venv/bin/activate`
        * For `fish`, run `source .venv/bin/activate.fish`
        * For `csh` or `tcsh`, run `source .venv/bin/activate.csh`
        * For `pwsh`, run `.venv/bin/Activate.ps1`
      </Tab>

      <Tab title="Windows">
        * For `cmd.exe`, run `.venv\Scripts\activate.bat`
        * For `PowerShell`, run `.venv\Scripts\Activate.ps1`
      </Tab>
    </Tabs>

    If you need to deactivate the virtual environment at any time, run the following command:

    ```bash theme={null}
    deactivate
    ```
  </Step>

  <Step title="Get the Unstructured API's workflow operations MCP Server's source code">
    The Unstructured API's workflow operations MCP Server is a Python package that provides an MCP server for the Unstructured API's workflow operations.
    To get the MCP server's source code, run the following command from the root of your project directory:

    ```bash theme={null}
    git clone https://github.com/Unstructured-IO/UNS-MCP.git
    ```
  </Step>

  <Step title="Install the Unstructured API's workflow operations MCP Server's code dependencies">
    From the root of your project directory, switch to the cloned Unstructured API's workflow operations MCP Server's source directory, and then use `uv` to install the MCP server's code dependencies, by running the following commands:

    ```bash theme={null}
    cd UNS-MCP
    uv sync
    ```
  </Step>

  <Step title="Install the CrewAI project's code dependencies">
    Switch back to the CrewAI project's root directory, and then use `uv` to install the CrewAI project's dependencies, by running the following commands:

    ```bash theme={null}
    cd ..
    uv add "crewai>=0.11.0" "crewai-tools[mcp]>=0.0.5" "pydantic>=2.11.0" python-dotenv
    ```
  </Step>

  <Step title="Add the CrewAI project's source code">
    In the `main.py` file in the CrewAI project's root directory, replace that file's contents with the following Python code. This code defines a set of
    CrewAI-compatible [agents](https://docs.crewai.com/concepts/agents) and [tasks](https://docs.crewai.com/concepts/tasks) that make up a
    multi-agent [crew](https://docs.crewai.com/concepts/crews). This code then uses the crew's agents to run the tasks in order to build an Unstructured ETL+ workflows
    from connector creation all the way through to workflow job completion:

    ```python theme={null}
    import os
    from typing import Optional
    from crewai import LLM, Agent, Crew, Task
    from crewai_tools import MCPServerAdapter
    from dotenv import load_dotenv
    from pydantic import BaseModel, Field
    from rich.pretty import pprint

    load_dotenv()

    class SourceConfigurationResult(BaseModel):
        source_id: Optional[str] = Field(
            default=None,
            description="The ID of the configured data source.",
        )
        source_type: Optional[str] = Field(
            default=None,
            description="The type of data source configured.",
        )
        source_name: Optional[str] = Field(
            default=None,
            description="The name of the configured data source.",
        )
        source_config: Optional[dict] = Field(
            default=None,
            description="The configuration details of the data source.",
        )

    class DestinationConfigurationResult(BaseModel):
        destination_id: Optional[str] = Field(
            default=None,
            description="The ID of the configured data destination.",
        )
        destination_type: Optional[str] = Field(
            default=None,
            description="The type of data destination configured.",
        )
        destination_name: Optional[str] = Field(
            default=None,
            description="The name of the configured data destination.",
        )
        destination_config: Optional[dict] = Field(
            default=None,
            description="The configuration details of the data destination.",
        )

    class WorkflowConfigurationResult(BaseModel):
        workflow_id: Optional[str] = Field(
            default=None,
            description="The ID of the created workflow.",
        )
        workflow_name: Optional[str] = Field(
            default=None,   
            description="The name of the created workflow.",
        )
        workflow_type: Optional[str] = Field(
            default=None,
            description="The type of workflow created.",
        )
        workflow_config: Optional[dict] = Field(
            default=None,
            description="The configuration details of the workflow.",
        )

    class JobConfigurationResult(BaseModel):
        job_id: Optional[str] = Field(
            default=None,
            description="The ID of the created job.",
        )

    class JobStatusResult(BaseModel):
        job_status: Optional[str] = Field(
            default=None,
            description="The status of the job.",
        )

    def main():

        with MCPServerAdapter(
            {"url": os.getenv("MCP_SERVER_URL", "http://127.0.0.1:8080/sse")},
        ) as tools:
            llm = LLM(model="anthropic/claude-3-opus-20240229", temperature=0.7, max_tokens=4096)

            source_agent = Agent(
                role="Source Configuration Specialist",
                goal="Configure and manage data sources for the MCP server",
                backstory="""You are an expert in data source configuration and management.
                You specialize in setting up and configuring various types of data sources
                including AWS S3, Google Drive, and other storage systems. You ensure
                proper configuration and validation of data sources.""",
                tools=tools,
                llm=llm,
                verbose=True,
            )

            destination_agent = Agent(
                role="Destination Configuration Specialist",
                goal="Configure and manage data destinations for the MCP server",
                backstory="""You are an expert in data destination configuration and management.
                You specialize in setting up and configuring various types of data destinations
                including AWS S3, Google Drive, and other storage systems. You ensure
                proper configuration and validation of data destinations.""",
                tools=tools,
                llm=llm,
                verbose=True,
            )

            workflow_agent = Agent(
                role="Workflow Configuration Specialist",
                goal="Configure and manage workflows for the MCP server",
                backstory="""You are an expert in workflow configuration and management.
                You specialize in setting up and configuring various types of workflows
                including AWS S3, Google Drive, and other storage systems. You ensure
                proper configuration and validation of workflows.""",
                tools=tools,
                llm=llm,
                verbose=True,
            )

            run_workflow_agent = Agent(
                role="Workflow Run Specialist",
                goal="Run workflows for the MCP server",
                backstory="""You are an expert in workflow run management.
                You specialize in running various types of workflows
                including AWS S3, Google Drive, and other storage systems. You ensure
                proper running of workflows.""",
                tools=tools,
                llm=llm,
                verbose=True,
            )

            job_agent = Agent(
                role="Job Management Specialist",
                goal="Manage jobs for the MCP server",
                backstory="""You are an expert in job management.
                You specialize in monitoring  various types of jobs
                proper monitoring of jobs.""",
                tools=tools,
                llm=llm,
                verbose=True,
            )

            source_task = Task(
                description=f"""Configure an S3 source with the following specifications:
                - Name: {os.getenv("S3_SOURCE_CONNECTOR_NAME")}
                - URI: {os.getenv("S3_SOURCE_URI")}
                - Recursive: true
                Ensure the source is properly configured and return the configuration details.""",
                agent=source_agent,
                expected_output="""A result containing:
                - source_id: The ID of the configured source
                - source_config: The configuration details
                - source_name: The name of the configured source
                - source_type: The type of data source configured
                """,
                output_pydantic=SourceConfigurationResult,
            )

            destination_task = Task(
                description=f"""Configure an S3 destination with the following specifications:
                - Name: {os.getenv("S3_DESTINATION_CONNECTOR_NAME")}
                - URI: {os.getenv("S3_DESTINATION_URI")}
                Ensure the destination is properly configured and return the configuration details.""",
                agent=destination_agent,
                expected_output="""A result containing:
                - destination_id: The ID of the configured destination
                - destination_config: The configuration details
                - destination_name: The name of the configured destination
                - destination_type: The type of data destination configured
                """,
                output_pydantic=DestinationConfigurationResult,
            )

            workflow_task = Task(
                description=f"""Create a workflow that uses the High Res partitioner and the following specifications:
                - Name: {os.getenv("WORKFLOW_NAME")}
                - Type: custom
                Use the source and destination IDs from the previous tasks to create the workflow.
                Ensure the workflow is properly configured to connect these endpoints.""",
                agent=workflow_agent,
                expected_output="""A result containing:
                - workflow_id: The ID of the created workflow
                - workflow_config: The configuration details
                - workflow_name: The name of the created workflow
                - workflow_type: The type of workflow created
                """,
                output_pydantic=WorkflowConfigurationResult,
                context=[source_task, destination_task],
            )

            run_workflow_task = Task(
                description=f"""Run a workflow.
                Use the workflow ID from the previous task to run the workflow.""",
                agent=run_workflow_agent,
                expected_output="""A result containing:
                - job_id: The ID of the running workflow
                """,
                output_pydantic=JobConfigurationResult,
                context=[workflow_task],
            )

            job_task = Task(
                description=f"""Monitor a job.
                Use the job ID from the previous task.
                Keep monitoring the job every 15 seconds until it is completed.
                Each time you check, report the current job status.
                When the job status is completed, stop reporting and return the final job status.""",
                agent=job_agent,
                expected_output="""A result containing:
                - job_status: The status of the job
                """,
                output_pydantic=JobStatusResult,
                context=[run_workflow_task],
            ) 

            crew = Crew(agents=[
                                   source_agent,
                                   destination_agent,
                                   workflow_agent,
                                   run_workflow_agent,
                                   job_agent
                                ], 
                        tasks=[
                                   source_task, 
                                   destination_task, 
                                   workflow_task, 
                                   run_workflow_task,
                                   job_task
                              ], 
                        verbose=True
                )

            result = crew.kickoff()
            pprint("Source Task Result:")
            pprint(result.tasks_output[0].pydantic)
            pprint("Destination Task Result:")
            pprint(result.tasks_output[1].pydantic)
            pprint("Workflow Task Result:")
            pprint(result.tasks_output[2].pydantic)
            pprint("Run Workflow Task Result:")
            pprint(result.tasks_output[3].pydantic)
            pprint("Job Task Result:")
            pprint(result.tasks_output[4].pydantic)

    if __name__ == "__main__":
        main()
    ```

    The preceding code does the following:

    1. Imports the necessary library modules for the rest of the code to use.
    2. Loads the environment variables that the code relies on from a `.env` file, which you will create in the next step.
    3. Defines the [Pydantic](https://docs.pydantic.dev/)-formatted models for the expected output of each task. These models format the tasks' output for consistent presentation.
    4. Defines the agents and tasks for the crew. These agents use their related tasks to create a source connector,
       a destination connector, and a workflow that uses these connectors, and then runs the newly created workflow as a job and reports the job's status.
    5. After the crew is finished, the results of each task are printed.
  </Step>

  <Step title="Create the .env file">
    Create an `.env` file in the root of your CrewAI project directory, and then add the following environment variables to the file:

    * `UNSTRUCTURED_API_KEY` - Your Unstructured API key.
    * `ANTHROPIC_API_KEY` - Your Anthropic API key.
    * `FIRECRAWL_API_KEY` - Your Firecrawl API key.
    * `AWS_KEY` - The AWS access key ID for the AWS IAM user that has access to the S3 bucket.
    * `AWS_SECRET` - The IAM user's AWS secret access key.
    * `S3_SOURCE_CONNECTOR_NAME` - Some display name for the S3 source connector.
    * `S3_SOURCE_URI` - The URI of the `input` folder in the S3 bucket.
    * `S3_DESTINATION_CONNECTOR_NAME` - The name of the S3 destination connector.
    * `S3_DESTINATION_URI` - The URI of the `output` folder in the S3 bucket.
    * `WORKFLOW_NAME` - Some display name for the workflow.

    For example, your `.env` file might look like this:

    ```bash theme={null}
    - UNSTRUCTURED_API_KEY=7F1...<redacted>hZV4
    - ANTHROPIC_API_KEY=sk-ant-api03-09b...<redacted>...wAA
    - FIRECRAWL_API_KEY=fc-af337...<redacted>...329
    - AWS_KEY=AKI...<redacted>...3DX
    - AWS_SECRET=end...<redacted>...Xxb
    - S3_SOURCE_CONNECTOR_NAME=My-MCP-S3-Source
    - S3_SOURCE_URI=s3://my-bucket/input
    - S3_DESTINATION_CONNECTOR_NAME=My-MCP-S3-Destination
    - S3_DESTINATION_URI=s3://my-bucket/output
    - WORKFLOW_NAME=My-MCP-S3-To-S3-Workflow
    ```
  </Step>

  <Step title="Unstructured API's workflow operations MCP Server">
    From the root of your project directory, switch to the Unstructured API's workflow operations MCP Server's source directory, and then use `make` to run the MCP server locally, by running the following commands:

    ```bash theme={null}
    cd UNS-MCP
    make sse-server
    ```

    <Note>If you do not have `make` available, see your operating system's documentation for installation instructions.</Note>

    The MCP server will start running at `http://127.0.0.1:8080/sse`.

    You must leave the MCP server running in your terminal or command prompt window while you run the CrewAI project.
  </Step>

  <Step title="Run the CrewAI project">
    1. In a separate terminal or command prompt window, from the root of your project directory, run the CrewAI project by running the following command:

       ```bash theme={null}
       uv run main.py
       ```

    2. The CrewAI project will run and create an Unstructured ETL+ workflow. You can see the crew's progress in the terminal or command prompt window where you ran the `uv run main.py` command.

    3. The crew's agents will create a source connector, a destination connector, and a workflow that uses these connectors.

    4. The crew's agents will then run the workflow as a job and report on the job's status.

    5. After the job is completed, the crew will report final information about the tasks that were completed, for example:

       ```bash theme={null}
       'Source Task Result:'
       SourceConfigurationResult(
       │   source_id='c26...<redacted>...83b',
       │   source_type='s3',
       │   source_name='My-MCP-S3-Source',
       │   source_config={
       │   │   'anonymous': False,
       │   │   'recursive': True,
       │   │   'remote_url': 's3://my-bucket/input',
       │   │   'endpoint_url': None,
       │   │   'key': '**********',
       │   │   'secret': '**********',
       │   │   'token': None
       │   }
       )
       'Destination Task Result:'
       DestinationConfigurationResult(
       │   destination_id='d57...<redacted>...ea4',
       │   destination_type='s3',
       │   destination_name='My-MCP-S3-Destination',
       │   destination_config={
       │   │   'anonymous': False,
       │   │   'remote_url': 's3://my-bucket/output',
       │   │   'endpoint_url': None,
       │   │   'key': '**********',
       │   │   'secret': '**********',
       │   │   'token': None
       │   }
       )
       'Workflow Task Result:'
       WorkflowConfigurationResult(
       │   workflow_id='ae5...<redacted>...cae',
       │   workflow_name='My-MCP-S3-To-S3-Workflow',
       │   workflow_type='custom',
       │   workflow_config={
       │   │   'name': 'My-MCP-S3-To-S3-Workflow',
       │   │   'workflow_type': 'custom',
       │   │   'source_id': 'c26...<redacted>...83b',
       │   │   'destination_id': 'd57...<redacted>...ea4',
       │   │   'workflow_nodes': [
       │   │   │   {
       │   │   │   │   'name': 'Partitioner',
       │   │   │   │   'type': 'partition',
       │   │   │   │   'subtype': 'unstructured_api',
       │   │   │   │   'settings': {
       │   │   │   │   │   'strategy': 'hi_res',
       │   │   │   │   │   'include_page_breaks': False,
       │   │   │   │   │   'pdf_infer_table_structure': False,
       │   │   │   │   │   'exclude_elements': None,
       │   │   │   │   │   'xml_keep_tags': False,
       │   │   │   │   │   'encoding': 'utf-8',
       │   │   │   │   │   'ocr_languages': ['eng'],
       │   │   │   │   │   'extract_image_block_types': ['image', 'table']
       │   │   │   │   }
       │   │   │   }
       │   │   ]
       │   }
       )
       'Run Workflow Task Result:'
       JobConfigurationResult(job_id='c3a5...<redacted>...bfa')
       'Job Task Result:'
       JobStatusResult(job_status='Completed')
       ```

    6. You can sign in to your Unstructured account to see the results:

       * On the sidebar, click **Connectors** to see the source and destination connectors.
       * On the sidebar, click **Workflows** to see the workflow.
       * On the sidebar, click **Jobs** to see the job.

    7. If you do not want to keep the MCP server running, you can stop it by pressing `Ctrl+C` in the terminal or command prompt window where you ran the `make sse-server` command.
  </Step>
</Steps>

## Learn more

* [Model Context Protocol (MCP) Hands-On Walkthrough for the Unstructured API's workflow operations](/examplecode/tools/mcp)
* [Unstructured API's workflow operations Reference](/api-reference/workflow/overview)
* [CrewAI Documentation](https://docs.crewai.com/)


# Databricks Volumes event triggers
Source: https://docs.unstructured.io/examplecode/tools/databricks-volumes-events



You can use Databricks Volumes events, such as uploading files to Databricks Volumes, to automatically run Unstructured ETL+ workflows
that rely on those Databricks Volumes as sources. This enables a no-touch approach to having Unstructured automatically process files as they are uploaded to Databricks Volumes.

This example shows how to automate this process by adding a custom job in Lakeflow Jobs for your Databricks workspace in
[AWS](https://docs.databricks.com/aws/jobs/), [Azure](https://learn.microsoft.com/azure/databricks/jobs/), or
[GCP](https://docs.databricks.com/gcp/jobs). This job runs
whenever a file upload event is detected in the specified Databricks Volume. This job uses a custom Databricks notebook to call the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow within your Unstructured account.

<Note>
  This example uses a custom job in Lakeflow Jobs and a custom Databricks notebookthat you create and maintain.
  Any issues with file detection, timing, or job execution could be related to your custom job or notebook,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  job's run logs first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

* A Databricks Volumes source connector in your Unstructured account. [Learn how](/pipelines/sources/databricks-volumes).

* Some available [destination connector](/pipelines/destinations/overview) in your Unstructured account.

* A workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

## Step 1: Create a notebook to run the Unstructured workflow

1. Sign in to the Databricks workspace within your Databricks account for AWS, Azure, or GCP that
   corresponds to the workspace you specified for your Databricks Volumes source connector.

2. On the sidebar, click **+ New > Notebook**.

3. Click the notebook's title and change it to something more descriptive, such as `Unstructured Workflow Runner Notebook`.

4. In the notebook's first cell, add the following code:

   ```python theme={null}
   !pip install requests
   ```

5. Click **Edit > Insert cell below**.

6. In this second cell, add the following code:

   ```python theme={null}
   import requests, os

   url = '<unstructured-api-url>' + '/workflows/<workflow-id>/run'

   # Option 1 (Recommended): Get your Unstructured API key from Databricks Secrets.
   api_key = dbutils.secrets.get(scope="<scope>", key="<key>")

   # Option 2: Get your Unstructured API key from an environment variable stored on 
   # the notebook's attached compute resource.
   api_key = os.getenv("UNSTRUCTURED_API_KEY")

   headers = {
       'accept': 'application/json',
       'content-type': 'application/json',
       'unstructured-api-key': api_key
   }

   json_data = {}

   try:
       response = requests.post(url, headers=headers, json=json_data)
       response.raise_for_status()
       print(f'Status Code: {response.status_code}')
       print('Response:', response.json())
   except Exception as e:
       print('An error occurred:', e)
   ```

7. Replace the placeholders in this second cell as follows:

   * Replace `<unstructured-api-url>` with the value of the **Unstructured API's workflow operations** field earlier from the requirements.
   * Replace `<workflow-id>` with the ID of the workflow that you want to run.
   * For your Unstructured API key, do one of the following:

     * (Recommended) If you want to use Databricks Secrets, replace `<scope>` and `<key>` with the scope and key names for the existing secret that you have already created in Databricks Secrets.
       Learn how to work with Databricks Secrets for
       [AWS](https://docs.databricks.com/aws/security/secrets/#secrets-overview),
       [Azure](https://learn.microsoft.com/azure/databricks/security/secrets/#secrets-overview), or
       [GCP](https://docs.databricks.com/gcp/security/secrets#secrets-overview).
     * If you want to use environment variables on the attached compute resource, set the `UNSTRUCTURED_API_KEY` to your Unstructured API key value. Learn how for
       [AWS](https://docs.databricks.com/aws/compute/configure#environment-variables),
       [Azure](https://learn.microsoft.com/azure/databricks/compute/configure#environment-variables), or
       [GCP](https://docs.databricks.com/gcp/compute/configure#environment-variables).

## Step 2: Create a job in Lakeflow Jobs to run the notebook

1. With your Databricks workspace still open from the previous step, on the sidebar, click **Jobs & Pipelines**.

2. Expand **Create new**, and then click **Job**.

3. Click the job's title and change it to something more descriptive, such as `Unstructured Workflow Runner Job`.

4. On the **Tasks** tab, enter some **Task name** such as `Run_Unstructured_Workflow_Runner_Notebook`.

5. With **Notebook** selected for **Type**, and with **Workspace** selected for **Source**, use the **Path** dropdown to select the notebook you created in the previous step.

6. For **Cluster**, select the cluster you want to use to run the notebook.

7. Click **Create task**.

8. In the **Job details** pane, under **Schedules & Triggers**, click **Add trigger**.

9. For **Trigger type**, select **File arrival**.

10. For **Storage location**, enter the path to the volume to monitor or, if you are monitoring a folder within that volume, the path to the folder. To get this path, do the following:

    a. On the sidebar, click **Catalog**.<br />
    b. In the list of catalogs, expand the catalog that contains the volume you want to monitor.<br />
    c. In the list of schemas (formerly known as databases), expand the schema that contains the volume you want to monitor.<br />
    d. Expand **Volumes**.<br />
    e. Click the volume you want to monitor.<br />
    f. On the **Overview** tab, copy the path to the volume you want to monitor or, if you are monitoring a folder within that volume, click the path to the folder and then copy the path to that folder.<br />

11. Click **Save**.

## Step 3: Trigger the job

1. With your Databricks workspace still open from the previous step, on the sidebar, click **Catalog**.
2. In the list of catalogs, expand the catalog that contains the volume that is being monitored.
3. In the list of schemas (formerly known as databases), expand the schema that contains the volume that is being monitored.
4. Expand **Volumes**.
5. Click the volume that is being monitored or, if you are monitoring a folder within that volume, click the folder.
6. Click **Upload to this volume**, and follow the on-screen instructions to upload a file to the volume or folder that is being monitored.

## Step 4: View trigger results

1. With your Databricks workspace still open from the previous step, on the sidebar, click **Jobs & Pipelines**.
2. On the **Jobs & pipelines** tab, click the name of the job you created earlier in Step 2.
3. On the **Runs** tab, wait until the current job run shows a **Status** of **Succeeded**.
4. In Unstructured Pipelines for your account, click **Jobs** on the sidebar.
5. In the list of jobs, click the newly running job for your workflow.
6. After the job status shows **Finished**, go to your destination location to see the results.

## Step 5 (Optional): Pause the trigger

To stop triggering the job, with your job in Lakeflow Jobs still open earlier from Step 4, in the **Job details** pane, under **Schedules & Triggers**, click **Pause**.


# Firecrawl
Source: https://docs.unstructured.io/examplecode/tools/firecrawl



[Firecrawl](https://firecrawl.dev/) is a tool that allows you to crawl websites and extract data from them.

You can use Unstructured to process the data that Firecrawl extracts from these websites. Unstructured turns this extracted
website data into a single set of standard [Unstructured document elements and metadata](/concepts/document-elements),
improving the accuracy, relevance, and performance of your RAG applications, agentic AI workflows, model fine tuning, and more.

This example provides a hands-on walkthrough of how to use Firecrawl with Unstructured. In this example, you use Python code to have Firecrawl
extract data from a ficticious website and then add the extracted data to an Amazon S3 bucket within your AWS account. From there, you use Unstructured Pipelines to set up an
ETL+ workflow that takes the extracted data from the S3 bucket, converts it into Unstructured document elements and metadata, adds
vector embeddings, and then sends the results over into a serverless index within a Pinecone vector store.

## Requirements

To complete this walkthrough, you must first have:

* An Unstructured account.
* A Firecrawl account and a Firecrawl API key for that account.
* An AWS account and a properly configured Amazon S3 bucket with bucket access credentials within that account.
* A Pinecone account and a Pinecone API key for that account.
* Python and the necessary development toolchain installed on your local development machine.

The following sections describe how to get these requirements.

### Unstructured account

Before you begin, you must have an Unstructured account and an Unstructured API key, as follows:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

### Firecrawl account and API key

Before you begin, you must have a Firecrawl account and a Firecrawl API key, as follows:

1. [Sign up or sign in to your Firecrawl account](https://www.firecrawl.dev/signin/signup).
2. After you sign in, [get your Firecrawl API key](https://www.firecrawl.dev/app/api-keys).

### Amazon S3 bucket with access permissions and credentials

This walkthrough uses an Amazon S3 bucket as the Unstructured ETL+ workflow's source. So, before you begin, you must have an AWS account, and within that account, a properly configured Amazon S3 bucket with bucket access credentials
consisting of an AWS access key ID and secret access key for the AWS IAM user that has access to the bucket.
Follow the S3 connector instructions to
[create and configure the bucket and get the bucket access credentials](/pipelines/sources/s3) if you do not already have this all set up. (In these
instructions, do not follow the directions to use Unstructured Pipelines to create the S3 source connector. You will do this later when you create the Unstructured ETL+ workflow.)

### Pinecone account and API key

Before you begin, you must have a Pinecone account and a Pinecone API key, as follows:

1. [Sign up](https://app.pinecone.io/?sessionType=signup) or [log in](https://app.pinecone.io/?sessionType=login) to your Pinecone account.
2. After you log in, [get your Pinecone API key](https://docs.pinecone.io/guides/projects/manage-api-keys#create-an-api-key).

### Python and development toolchain installation

Before you can start coding on your local machine, you must install Python, and you should also install a Python package and project manager to manage your project's Firecrawl, AWS, Pinecone, and other code dependencies.
This walkthrough uses the popular Python package and project manager [uv](https://docs.astral.sh/uv/) (although `uv` is not required to use Firecrawl, AWS, or Pinecone with Python).

<Steps>
  <Step title="Install uv">
    To install `uv`, run one of the following commands, depending on your operating system:

    <Tabs>
      <Tab title="macOS, Linux">
        To use `curl` with `sh`:

        ```bash theme={null}
        curl -fsSL https://get.uv.dev | bash
        ```

        To use `wget` with `sh` instead:

        ```bash theme={null}
        wget -qO- https://astral.sh/uv/install.sh | sh
        ```
      </Tab>

      <Tab title="Windows">
        To use PowerShell with `irm` to download the script and run it with `iex`:

        ```powershell theme={null}
        powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
        ```
      </Tab>
    </Tabs>

    If you need to install `uv` by using other approaches such as PyPI, Homebrew, or WinGet,
    see [Installing uv](https://docs.astral.sh/uv/getting-started/installation/).
  </Step>

  <Step title="Install Python">
    <Note>This walkthrough should work with Python 3.9 or later and has been tested with Python 3.12.</Note>

    `uv` will detect and use Python if you already have it installed.
    To view a list of installed Python versions, run the following command:

    ```bash theme={null}
    uv python list
    ```

    If, however, you do not already have Python installed, you can install a version of Python for use with `uv`
    by running the following command. For example, this command installs Python 3.12 for use with `uv`:

    ```bash theme={null}
    uv python install 3.12
    ```
  </Step>

  <Step title="Create the project directory">
    Switch to the directory on your local development machine where you want to create the project directory for this walkthrough.
    This example creates a project directory named `firecrawl_unstructured_demo` within your current working directory and then
    switches to this new project directory:

    ```bash theme={null}
    mkdir firecrawl_unstructured_demo
    cd firecrawl_unstructured_demo
    ```
  </Step>

  <Step title="Intiialize the project">
    From within the new project directory, use `uv` to initialize the project by running the following command:

    ```bash theme={null}
    uv init
    ```
  </Step>

  <Step title="Create a venv virtual environment">
    To isolate and manage your project's code dependencies, you should create a virtual environment. This walkthrough uses
    the popular Python virtual environment manager [venv](https://docs.python.org/3/library/venv.html) (although `venv` is not required to use CrewAI or the Unstructured API's workflow operations MCP Server).
    From the root of your project directory, use `uv` to create a virtual environment with `venv` by running the following command:

    ```bash theme={null}
    # Create the virtual environment by using the current Python version:
    uv venv

    # Or, if you want to use a specific Python version:
    uv venv --python 3.12
    ```
  </Step>

  <Step title="Activate the virtual environment">
    To activate the `venv` virtual environment, run one of the following commands from the root of your project directory:

    <Tabs>
      <Tab title="macOS, Linux">
        * For `bash` or `zsh`, run `source .venv/bin/activate`
        * For `fish`, run `source .venv/bin/activate.fish`
        * For `csh` or `tcsh`, run `source .venv/bin/activate.csh`
        * For `pwsh`, run `.venv/bin/Activate.ps1`
      </Tab>

      <Tab title="Windows">
        * For `cmd.exe`, run `.venv\Scripts\activate.bat`
        * For `PowerShell`, run `.venv\Scripts\Activate.ps1`
      </Tab>
    </Tabs>

    If you need to deactivate the virtual environment at any time, run the following command:

    ```bash theme={null}
    deactivate
    ```
  </Step>

  <Step title="Install the Firecrawl Python SDK">
    You will use the Firecrawl Python SDK later to extract data from a website.

    Use `uv` to install the Firecrawl Python SDK package, by running the following command:

    ```bash theme={null}
    uv add firecrawl-py
    ```
  </Step>

  <Step title="Install the AWS SDK for Python">
    You will use the AWS SDK for Python later to have Firecrawl add the extracted data to your Amazon S3 bucket.

    1. Use `uv` to install the AWS SDK for Python package, by running the following command:

       ```bash theme={null}
       uv add boto3
       ```

    2. To enable the AWS SDK for Python to authenticate with your AWS account, create a file named `credentials` in the `~/.aws/` directory for macOS or Linux, or the `<drive>:\Users\<username>\.aws\` directory for Windows. Then add the
       AWS access key ID and secret access key of the AWS IAM user that has access to your Amazon S3 bucket, and the [short code for the AWS Region](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html) of the bucket (for example, `us-east-1`), to the `credentials` file.

       In the following `credentials` file example, replace the following placeholders:

       * Replace `<your-access-key-id>` with the AWS access key ID of the AWS IAM user that has access to the bucket.
       * Replace `<your-secret-access-key>` with the secret access key for the related access key ID.
       * Replace `<the-aws-short-region-code-for-your-bucket>` with the short code for the AWS Region of the bucket.

       ```text theme={null}
       [default]
       aws_access_key_id = <your-access-key-id>
       aws_secret_access_key = <your-secret-access-key>
       region = <the-aws-short-region-code-for-your-bucket>
       ```
  </Step>

  <Step title="Install the Pinecone Python SDK">
    You will use the Pinecone Python SDK later to create the Pinecone serverless index and, in an optional step, to help enable RAG chat on the data that will be storedin the index.

    Use `uv` to install the Pinecone Python SDK package, along with the `grpc` extra to enable the programmatic creation of a Pinecone serverless index later, by running the following command:

    ```bash theme={null}
    uv add "pinecone[grpc]"
    ```
  </Step>
</Steps>

## Step 1: Extract the website data and add the data to the bucket

In this step, you use Firecrawl to extract data from a website and then add the extracted data to the Amazon S3 bucket.

<Steps>
  <Step title="Add your Firecrawl API key to your project">
    1. Set your Firecrawl API key in your project as an environment variable for your code to use. To do this:

       a. In the root of your code project, create a file named `.env`.<br />
       b. Add the following environment variable to this `.env` file, and replace
       `<your-firecrawl-api-key>` with your Firecrawl API key:<br />

       ```text theme={null}
       FIRECRAWL_API_KEY=<your-firecrawl-api-key>
       ```

    2. To enable your Firecrawl code to access the environment variable, use `uv` to install the `dotenv` package by running the following command:

       ```bash theme={null}
       uv add dotenv
       ```

    3. In the `.gitignore` file in the root of your project, add the following line, to help prevent
       accidentally checking in your Firecrawl API key (or anything else in the `.env` file) into any shared code repositories later:

       ```text theme={null}
       .env
       ```
  </Step>

  <Step title="Create the Python script to extract the website data">
    1. In the `.env` file, add the following line, which defines an environment variable representing the base URL
       of the website to crawl. This walkthrough uses a website named [Books to Scrape](https://books.toscrape.com/) that contains ficticious data (although you can use any accessible website you want):

       ```text theme={null}
       FIRECRAWL_BASE_URL=https://books.toscrape.com/
       ```

    2. Also in the `.env` file, add the following line, which defines an environment variable representing the name of the target Amazon S3 bucket to
       have Firecrawlstore the website crawl results in. Replace `<the-name-of-your-bucket>` with the name of the bucket:

       ```text theme={null}
       FIRECRAWL_S3_BUCKET_NAME=<the-name-of-your-bucket>
       ```

       <Note>Enter only the name of the bucket. Do not prefix the bucket's name with `s3://`. Do not include any trailing slash (`/`) after the bucket name.</Note>

    3. Create a file named `firecrawl_extract.py` in the root of your project directory, and add the following code to it:

       ```python theme={null}
       from firecrawl import FirecrawlApp, ScrapeOptions
       from dotenv import load_dotenv
       import os, boto3, json
       from datetime import datetime

       load_dotenv()

       def save_to_s3(crawl_result, bucket_name, prefix):
           s3 = boto3.client("s3")
           timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")

           # Save the full crawl results to the bucket as a single JSON file.
           full_results_key = f"{prefix}/{timestamp}/full_results.json"
           s3.put_object(
               Bucket=bucket_name,
               Key=full_results_key,
               Body=json.dumps(crawl_result.model_dump(), indent=2, default=str),
           )

           # Save individually crawled pages to the bucket as HTML files.
           for idx, page in enumerate(crawl_result.data):
               if page.html is not None:
                   page_key = f"{prefix}/{timestamp}/pages/{idx}.html"
                   s3.put_object(Bucket=bucket_name, Key=page_key, Body=page.html)
                   print(f"Successfully saved {idx}.html to {bucket_name}/{page_key}")
           print(f"Successfully saved {len(crawl_result.data)} pages to {bucket_name}/{full_results_key}")

       def main():
           app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))

           result = app.crawl_url(
               url=os.getenv("FIRECRAWL_BASE_URL"),
               limit=5, # Maximum number of pages to crawl.
               scrape_options=ScrapeOptions(
                   formats=["html"], # Save the crawled pages as HTML files.
                   waitFor=1000, # Wait for one second before crawling the currentpage.
                   timeout=10000 # Move on to the next page if no response within 10 seconds.
               )
           )
           
           save_to_s3(
               crawl_result=result,
               bucket_name=os.getenv("FIRECRAWL_S3_BUCKET_NAME"),
               prefix="crawls"
           )

       if __name__ == "__main__":
           main()
       ```

       The preceding code does the following:

       * Loads from your `.env` file your Firecrawl API key, the base URL for Firecrawl to use for website crawling, and the S3 bucket name for Firecrawl to send the website crawl results to.
       * Calls a function named `main` that uses Firecrawl to crawl the website.
       * The `main` function then calls the `save_to_s3` function, which adds the website crawl results to the S3 bucket.

       You can change the where the website crawl results are saved in your S3 bucket by changing the `full_results_key` and `page_key` variables. For example, you might want to save the
       website crawl results in a single folder and then keep overwriting those results with new results as they come in, instead of adding new results to separate subfolders.

       You can also change the number of crawled pages by changing the `limit` argument, outputting the results as Markdown instead of HTML, and so on. For more
       information, see the [Firecrawl Python SDK documentation](https://docs.firecrawl.dev/sdks/python).
  </Step>

  <Step title="Run the script to extract the data">
    From the root of your project, use `uv` to run the script to extract the data from the website by running the following command:

    ```bash theme={null}
    uv run firecrawl_extract.py
    ```

    After the script runs, you should see the following output, where `<your-bucket-name>` is the name of your bucket, and `<timestamp>` is the timestamp generated by the script:

    ```bash theme={null}
    Successfully saved 0.html to <your-bucket-name>/crawls/<timestamp>/pages/0.html
    Successfully saved 1.html to <your-bucket-name>/crawls/<timestamp>/pages/1.html
    Successfully saved 2.html to <your-bucket-name>/crawls/<timestamp>/pages/2.html
    Successfully saved 3.html to <your-bucket-name>/crawls/<timestamp>/pages/3.html
    Successfully saved 4.html to <your-bucket-name>/crawls/<timestamp>/pages/4.html
    Successfully saved 5 pages to <your-bucket-name>/crawls/<timestamp>/full_results.json
    ```

    To verify that the data was successfully added, open your AmazonS3 console
    and look in the `crawls` folder within your bucket.
  </Step>
</Steps>

## Step 2: Create a Pinecone serverless index

In this step, you create a Pinecone serverless index that Unstructured will use to store its results based on the
extracted website data that Firecrawl sends to your Amazon S3 bucket.

<Steps>
  <Step title="Add your Pinecone API key to your project">
    Set your Pinecone API key as an environment variable for your code to use. To do this, add the following environment variable to your `.env` file, and replace
    `<your-pinecone-api-key>` with your Pinecone API key:

    ```text theme={null}
    PINECONE_API_KEY=<your-pinecone-api-key>
    ```
  </Step>

  <Step title="Add the Pinecone index name to your project">
    Add the following line to your `.env` file, and replace `<the-name-of-your-index>` with the name of the serverlessindex you want to create:

    ```text theme={null}
    PINECONE_INDEX_NAME=<the-name-of-your-index>
    ```
  </Step>

  <Step title="Create the Python script to create the Pinecone serverless index">
    Create a file named `pinecone_create_index.py` in the root of your project directory and add the following code to it:

    ```python theme={null}
    from pinecone.grpc import PineconeGRPC as Pinecone
    from pinecone import ServerlessSpec
    import os
    from dotenv import load_dotenv

    load_dotenv()

    pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))

    pc.create_index(
        name=os.getenv("PINECONE_INDEX_NAME"),
        vector_type="dense",
        dimension=3072,
        metric="cosine",
        spec=ServerlessSpec(
            cloud="aws",
            region="us-east-1"
        )
    )
    ```

    The preceding code creates in your Pinecone account a serverless index with the specified name for the specified cloud provider and within the specified region for that provider.
    The index is configured to accept vectors with 3072 dimensions with a cosine metric and a dense vector type.
  </Step>

  <Step title="Run the script to create the index">
    From the root of your project, use `uv` to run the script to create the Pinecone serverless index by running the following command:

    ```bash theme={null}
    uv run pinecone_create_index.py
    ```

    After the script runs, to verify that the index was successfully added, log in to your Pinecone account. Then open your default project,
    and click **Indexes** under **Database** in the sidebar to view the new index that was added.
  </Step>
</Steps>

## Step 3: Set up and run the Unstructured ETL+ workflow

In this step, you set up and run the Unstructured ETL+ workflow that will take the extracted website data from your Amazon S3 bucket, process it to turn it into
Unstructured elements and metadata, add vector embeddings to the elements, and then add all of this information to your Pinecone serverless index.

<Steps>
  <Step title="Create the source connector">
    1. If you are not already signed in, sign in to your Unstructured account.
    2. On the sidebar, click **Connectors**.
    3. With **Sources** selected, **New**.
    4. Enter some name for the source connector, such as `s3-firecrawl-source`.
    5. For **Provider**, click **Amazon S3**.
    6. Click **Continue**.
    7. For **Bucket URI**, enter `s3://<your-bucket-name>/crawls/<timestamp>/pages/`, replacing `<your-bucket-name>` with the name of your bucket, and `<timestamp>` with the timestamp generated by the script in the previous step..
    8. For **Authentication Method**, select **Secret**.
    9. Enter your AWS access key ID and secret access key.
    10. Click **Continue**.
    11. Click **Save and Test**.
  </Step>

  <Step title="Create the destination connector">
    1. On the sidebar, click **Connectors**.
    2. With **Destinations** selected, **New**.
    3. Enter some name for the destination connector, such as `pinecone-firecrawl-destination`.
    4. For **Provider**, click **Pinecone**.
    5. Click **Continue**.
    6. For **Index Name**, enter the name of the Pinecone index you created earlier, such as `firecrawl-dense-index`.
    7. Leave **Namespace** blank. Unstructured will automatically use the default namespace in the index.
    8. Leave **Batch Size** set to the default value of `50`.
    9. For **API Key**, enter your Pinecone API key.
    10. Click **Save and Test**.
  </Step>

  <Step title="Create the workflow">
    1. On the sidebar, click **Workflows**.
    2. Click **New Workflow**.
    3. With **Build it Myself** selected, click **Continue**.
    4. In the workflow designer, click the **Source** workflow node.
    5. In the node settings pane, on the **Connectors** tab, select the name of the source connector you created earlier, such as `s3-firecrawl-source`.
    6. Click the **Destination** workflow node.
    7. In the node settings pane, on the **Connectors** tab, select the name of the destination connector you created earlier, such as `pinecone-firecrawl-destination`.
    8. Between the **Partitioner** and **Destination** nodes, click the **+** (add node) button.
    9. Click **Transform > Embedder**.
    10. Click the **Embedder** workflow node.
    11. In the **Settings** pane, under **Azure OpenAI**, select **text-embedding-3-large**.
    12. Click any blank space in the workflow designer.
    13. In the workflow settings pane, click **Details**.
    14. For **Name**, click the pencil (edit) icon and then enter some name for the workflow, such as `firecrawl-s3-to-pinecone-workflow`.
        Then click the checkmark (save) icon.
    15. Switch **Active** to on.
    16. Click **Save**.
  </Step>

  <Step title="Run the workflow as a job">
    1. On the sidebar, click **Workflows**.
    2. Next to the name of the workflow you created earlier, click **Run**.
  </Step>

  <Step title="Monitor the job">
    1. On the sidebar, click **Jobs**.
    2. Click the name of the workflow you created earlier.
    3. Monitor the job's progress in the job details pane until the status changes to **Finished**.
  </Step>

  <Step title="View the results">
    1. One way to view the results is to create a file named `pinecone_fetch_from_index.py` in the root of your project directory and add the following code to it:

       ```python theme={null}
       from pinecone import Pinecone
       import os
       from dotenv import load_dotenv

       load_dotenv()

       pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
       index = pc.Index(os.getenv("PINECONE_INDEX_NAME"))
       ids = []

       # The list_paginated() method returns a dict with 'vectors' key
       paginated_results = index.list_paginated(namespace="default")
       for page in paginated_results['_data_store']['vectors']:
           ids.append(page['id'])

       response = index.fetch(
           ids=ids,
           namespace="default"
       )

       for vector_id, vector_data in response["vectors"].items():
           print(f"ID: {vector_id}\nMetadata: {vector_data.get('metadata')}\n\n")
       ```

    2. From the root of your project, use `uv` to run the script as follows:

       ```bash theme={null}
       uv run pinecone_fetch_from_index.py
       ```

       The output will show each record's ID and metadata. For brevity and performance,
       the records' vector embeddings are not included in the output.
  </Step>
</Steps>

## Additional resources

* [Mastering Firecrawl's Crawl Endpoint: A Complete Web Scraping Guide](https://www.firecrawl.dev/blog/mastering-the-crawl-endpoint-in-firecrawl)


# Google Cloud Storage event triggers
Source: https://docs.unstructured.io/examplecode/tools/gcs-events



You can use Google Cloud Storage events, such as adding new files to—or updating existing files within—Google Cloud Storage buckets, to automatically run Unstructured ETL+ workflows
that rely on those buckets as sources. This enables a no-touch approach to having Unstructured automatically process new and updated files in Google Cloud Storage as they are added or updated.

This example shows how to automate this process by adding a custom [Google Apps Script](https://developers.google.com/apps-script) project in your Google account. This project runs
a script on a regular time interval. This script automatically checks for new or updated files within the specified Google Cloud Storage bucket. If the script
detects at least one new or updated file, it then calls the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow in your Unstructured account.

<Note>
  This example uses a custom Google Apps Script that you create and maintain.
  Any issues with file detection, timing, or script execution could be related to your custom script,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  script's execution logs first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

* A Google Cloud Storage source connector in your Unstructured account. [Learn how](/pipelines/sources/google-cloud).

* Some available [destination connector](/pipelines/destinations/overview) in your Unstructured account.

* A workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

* An OAuth 2.0 client ID and client secret to call the Google API, as follows:

  1. Sign in to your [Google Cloud account](https://cloud.google.com).
  2. Go to the [Google Cloud APIs dashboard](https://console.cloud.google.com/apis/dashboard).
  3. Click **+ Enable APIs and services**.
  4. In the **Search for APIs & Services** box, enter `Cloud Storage API`.
  5. In the list of search results, click **Cloud Storage API**.
  6. Make sure that **API Enabled** is shown. If not, click **Enable**.
  7. Go to your [Google Cloud console welcome page](https://console.cloud.google.com/welcome).
  8. In the **Search (/) for resources, docs, products, and more** box, enter `Credentials`.
  9. Click **Credentials (APIs & Services)**.
  10. Click **+ Create credentials > OAuth client ID**.
  11. For **Application type**, select **Web application**.
  12. (Optional) Enter some non-default **Name** for this OAuth 2.0 client to be shown in the list of created clients in your Google Cloud Console.
  13. Click **Create**.
  14. After the OAuth client is created, click **Download JSON** to save the client ID and client secret values to a JSON file on your local
      machine. Store this JSON file in a secure location.

## Step 1: Create the Google Apps Script project

1. Go to [http://script.google.com/](http://script.google.com/).
2. Click **+ New project**.
3. Click the new project's default name (such as **Untitled project**), and change it to something more descriptive, such as **Unstructured Scripts for GCS**.

## Step 2: Add the script

1. With the project still open, on the sidebar, click the **\< >** (**Editor**) icon.

2. In the **Files** tab, click **Code.gs**.

3. Replace the contents of the `Code.gs` file with the following code instead:

   ```javascript theme={null}
   // Configure the OAuth2 service.
   function getOAuthService() {
     return OAuth2.createService('GCS')
       .setAuthorizationBaseUrl('https://accounts.google.com/o/oauth2/auth')
       .setTokenUrl('https://oauth2.googleapis.com/token')
       .setClientId(CLIENT_ID)
       .setClientSecret(CLIENT_SECRET)
       .setCallbackFunction('authCallback')
       .setPropertyStore(PropertiesService.getUserProperties())
       .setScope(OAUTH_SCOPE);
   }

   // OAuth2 callback function.
   function authCallback(request) {
     const service = getOAuthService();
     const isAuthorized = service.handleCallback(request);
     return HtmlService.createHtmlOutput(isAuthorized ? 'Success!' : 'Denied');
   }

   // Get a valid access token (refreshes automatically if expired).
   function getAccessToken() {
     const service = getOAuthService();
     if (!service.hasAccess()) {
       const authorizationUrl = service.getAuthorizationUrl();
       Logger.log('Open the following URL and re-run the script: %s', authorizationUrl);
       throw new Error('Authorization required. Open the URL in the log.');
     }
     return service.getAccessToken();
   }

   // Main function: checks for new or updated files in the bucket.
   function checkForNewOrUpdatedGCSFiles() {
     const thresholdMillis = 5 * 60 * 1000; // 5 minutes.
     const now = new Date();

     // Get (and refresh if needed) the access token.
     const accessToken = getAccessToken();

     // List objects in the bucket.
     const apiUrl = `https://storage.googleapis.com/storage/v1/b/${BUCKET_PATH}/o`;
     const response = UrlFetchApp.fetch(apiUrl, {
       method: 'get',
       headers: {
         'Authorization': 'Bearer ' + accessToken,
         'Accept': 'application/json'
       }
     });
     const data = JSON.parse(response.getContentText());
     const files = data.items || [];

     for (let i = 0; i < files.length; i++) {
       const file = files[i];
       const fileName = file.name;
       const created = new Date(file.timeCreated);
       const updated = new Date(file.updated);

       const millisSinceCreated = now - created;
       const createdWithinThreshold = millisSinceCreated < thresholdMillis;
       const millisSinceUpdated = now - updated;
       const updatedWithinThreshold = millisSinceUpdated < thresholdMillis;

       console.log('File Name:                       ' + fileName);
       console.log('Created:                         ' + created);
       console.log('Last updated:                    ' + updated);
       console.log('Milliseconds since created:      ' + millisSinceCreated);
       console.log('Milliseconds since last updated: ' + millisSinceUpdated);
       console.log('Created within threshold of ' + thresholdMillis + ' milliseconds? ' + createdWithinThreshold);
       console.log('Updated within threshold of ' + thresholdMillis + ' milliseconds? ' + updatedWithinThreshold);
       console.log('-----');

       if (createdWithinThreshold || updatedWithinThreshold) {
         // Trigger your workflow.
         UrlFetchApp.fetch(UNSTRUCTURED_API_URL, {
           method: 'post',
           headers: {
             'accept': 'application/json',
             'unstructured-api-key': UNSTRUCTURED_API_KEY
           }
         });
         console.log('At least one file created or updated within threshold of ' + thresholdMillis + ' milliseconds.');
         console.log('Unstructured workflow request sent to ' + UNSTRUCTURED_API_URL);
         return;
       }
     }
     console.log('No files created or updated within threshold of ' + thresholdMillis + ' milliseconds. No Unstructured workflow request sent.');
   }

   ```

4. Click the **Save project to Drive** button.

## Step 3: Customize the script for your workflow

1. With the project still open, on the **Files** tab, click the **Add a file** button, and then click **Script**.

2. Name the new file `Constants`. The `.gs` extension is added automatically.

3. Replace the contents of the `Constants.gs` file with the following code instead:

   ```javascript theme={null}
   const BUCKET_PATH = '<bucket-path>';
   const UNSTRUCTURED_API_URL = '<unstructured-api-url>' + '/workflows/<workflow-id>/run';
   const UNSTRUCTURED_API_KEY = '<unstructured-api-key>';
   const CLIENT_ID = '<client-id>';
   const CLIENT_SECRET = '<client-secret>';
   const OAUTH_SCOPE = 'https://www.googleapis.com/auth/devstorage.read_only'; // Or .read_write or .full_control
   ```

   Replace the following placeholders:

   * Replace `<bucket-path>` with the path to your Google Cloud Storage bucket. This is the same path that you specified
     when you created your Google Cloud Storage source connector in your Unstructured account. Do not include the `gs://` prefix here.
   * Replace `<unstructured-api-url>` with your Unstructured API URL value.
   * Replace `<workflow-id>` with the ID of your Unstructured workflow.
   * Replace `<unstructured-api-key>` with your Unstructured API key value.
   * Replace `<client-id>` with your OAuth 2.0 client ID value.
   * Replace `<client-secret>` with your OAuth 2.0 client secret value.

4. Click the disk (**Save project to Drive**) icon.

## Step 4: Generate an initial OAuth 2.0 access token

1. On the sidebar, click the gear (**Project Settings**) icon.
2. In the **IDs** area, next to **Script ID**, click **Copy** to copy the script's ID value to your system's clipboard.
3. In a separate tab in your web browser, open your [Google Cloud Console welcome page](https://console.cloud.google.com/welcome).
4. In the **Search (/) for resources, docs, products, and more** box, enter `Credentials`.
5. Click **Credentials (APIs & Services)**.
6. In the **OAuth 2.0 client IDs** list, click the link for the client ID that you created earlier in the requirements.
7. Under **Authorized redirect URIs**, click **Add URI**.
8. In the **URIs 1** box, enter `https://script.google.com/macros/d/<script-id>/usercallback`, replacing `<script-id>` with the script's ID value that you copied earlier.
9. Click **Save**.
10. On the original tab in your web browser, with the Google Apps Script project still open to the **Constants.gs** file, on the sidebar, next to **Libraries**, click the **+** (**Add a library**) icon.
11. For **Script ID**, enter `1B7FSrk5Zi6L1rSxxTDgDEUsPzlukDsi4KGuTMorsTQHhGBzBkMun4iDF`, and then click **Look up**.
12. For **Version**, make sure the largest number is selected.
13. Click **Add**.
14. In the sidebar, click the **Code.gs** file to open it.
15. In the file's top navigation bar, select **getAccessToken**.
16. Click the **Run** icon.
17. In the **Execution log** area, next to the message `Open the following URL and re-run the script`, copy the entire URL into
    a separate tab in your web browser and then browse to that URL.
18. When prompted, click **Review permissions**, and follow the on-screen instructions to grant the necessary permissions.

## Step 5: Create the script trigger

1. On the original tab in your web browser, with the Google Apps Script project still open, on the sidebar, click the alarm clock (**Triggers**) icon.

2. Click the **+ Add Trigger** button.

3. Set the following values:

   * For **Choose which function to run**, select `checkForNewOrUpdatedGCSFiles`.

   * For **Choose which deployment should run**, select **Head**.

   * For **Select event source**, select **Time-driven**.

   * For **Select type of time based trigger**, select **Minutes timer**.

   * For **Select minute interval**, select **Every 5 minutes**.

     <Note>
       If you change **Minutes timer** or **Every 5 minutes** to a different interval, you should also go back and change the number `5` in the following
       line of code in the `checkForNewOrUpdatedFiles` function. Change the number `5` to the number of minutes that correspond to the alternate interval you
       selected:

       ```javascript theme={null}
       const thresholdMillis = 5 * 60 * 1000;
       ```
     </Note>

   * For **Failure notification settings**, select an interval such as immediately, hourly, or daily.

4. Click **Save**.

## Step 6: View trigger results

1. With the Google Apps Script project still open, on the sidebar, click the three lines (**Executions**) icon.

2. As soon as the first script execution completes, you should see a corresponding message appear in the **Executions** list. If the **Status** column shows
   **Completed**, then keep going with this procedure.

   If the **Status** column shows **Failed**, expand the message to
   get any details about the failure. Fix the failure, and then wait for the next script execution to complete.

3. When the **Status** column shows **Completed** then, in your Unstructured account's user interface, click **Jobs** on the sidebar to see if a new job
   is running for that worklow.

   If no new job is running for that workflow, then add at least one new file to—or update at least one existing file within—the Google Cloud Storage bucket,
   within 5 minutes of the next script execution. After the next script execution, check the **Jobs** list again.

## Step 7 (Optional): Delete the trigger

1. To stop the script from automatically executing on a regular basis, with the Google Apps Script project still open, on the sidebar, click the alarm clock (**Triggers**) icon.
2. Rest your mouse pointer on the trigger you created in Step 5.
3. Click the ellipsis (three dots) icon, and then click **Delete trigger**.


# Google Drive event triggers
Source: https://docs.unstructured.io/examplecode/tools/google-drive-events



You can use Google Drive events, such as adding new files to—or updating existing files in—Google Drive shared folders or shared drives, to automatically run Unstructured ETL+ workflows
that rely on those folders or drives as sources. This enables a no-touch approach to having Unstructured automatically process new and updated files in Google Drive as they are added or updated.

This example shows how to automate this process by adding a custom [Google Apps Script](https://developers.google.com/apps-script) project in your Google account. This project runs
a script on a regular time interval. This script automatically checks for new or updated files within the specified Google Drive shared folder or shared drive. If the script
detects at least one new or updated file, it then calls the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow in your Unstructured account.

<Note>
  This example uses a custom Google Apps Script that you create and maintain.
  Any issues with file detection, timing, or script execution could be related to your custom script,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  script's execution logs first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

* A Google Drive source connector in your Unstructured account. [Learn how](/pipelines/sources/google-drive).

* Some available [destination connector](/pipelines/destinations/overview) in your Unstructured account.

* A workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

## Step 1: Create the Google Apps Script project

1. Sign in to your Google account.
2. Go to [http://script.google.com/](http://script.google.com/).
3. Click **+ New project**.
4. Click the new project's default name (such as **Untitled project**), and change it to something more descriptive, such as **Unstructured ETL Scripts**.

## Step 2: Add the script

1. With the project still open, on the sidebar, click the **\< >** (**Editor**) icon.

2. In the **Files** tab, click **Code.gs**.

3. Replace the contents of the `Code.gs` file with the following code instead:

   ```javascript theme={null}
   function checkForNewOrUpdatedFiles() {
     const folder = DriveApp.getFolderById(FOLDER_ID);
     const files = folder.getFiles();
     const now = new Date();
     const thresholdMillis = 5 * 60 * 1000; // 5 minutes (adjust as needed).
     
     while (files.hasNext()) {
       const file = files.next();
       const created = file.getDateCreated();
       const lastUpdated = file.getLastUpdated();
       const fileName = file.getName();

       // Calculate time differences.
       const millisSinceCreated = now - created;
       const createdWithinThreshold = millisSinceCreated < thresholdMillis;
       const millisSinceUpdated = now - lastUpdated;
       const updatedWithinThreshold = millisSinceUpdated < thresholdMillis;

       // Log file details and calculations.
       console.log('File Name:                       ' + fileName);
       console.log('Created:                         ' + created);
       console.log('Last updated:                    ' + lastUpdated);
       console.log('Milliseconds since created:      ' + millisSinceCreated);
       console.log('Milliseconds since last updated: ' + millisSinceUpdated);
       console.log('Created within threshold of ' + thresholdMillis + ' milliseconds? ' + createdWithinThreshold);
       console.log('Updated within threshold of ' + thresholdMillis + ' milliseconds? ' + updatedWithinThreshold);
       console.log('-----')

       // If at least one file was created or updated within the last 5 minutes...
       if ((createdWithinThreshold) || (updatedWithinThreshold)) {
         // ...then make the HTTP POST request.
         UrlFetchApp.fetch(UNSTRUCTURED_API_URL, {
           method: 'post',
           headers: {
             'accept': 'application/json',
             'unstructured-api-key': UNSTRUCTURED_API_KEY
           }
         });
         // Then stop the script after the first fetch (no need to check any more files).
         console.log('At least one file created or updated within threshold of ' + thresholdMillis + ' milliseconds.')
         console.log('Unstructured workflow request sent to ' + UNSTRUCTURED_API_URL)
         return;
       }
     }
     console.log('No files created or updated within threshold of ' + thresholdMillis + ' milliseconds. No Unstructured workflow request sent.')
   }
   ```

4. Click the **Save project to Drive** button.

## Step 3: Customize the script for your workflow

1. With the project still open, on the **Files** tab, click the **Add a file** button, and then click **Script**.

2. Name the new file `Constants`. The `.gs` extension is added automatically.

3. Replace the contents of the `Constants.gs` file with the following code instead:

   ```javascript theme={null}
   const FOLDER_ID = '<folder-id>';
   const UNSTRUCTURED_API_URL = '<unstructured-api-url>' + '/workflows/<workflow-id>/run';
   const UNSTRUCTURED_API_KEY = '<unstructured-api-key>';
   ```

   Replace the following placeholders:

   * Replace `<folder-id>` with the ID of your Google Drive shared folder or shared drive. This is the same ID that you specified
     when you created your Google Drive source connector in your Unstructured account.
   * Replace `<unstructured-api-url>` with your Unstructured API URL value.
   * Replace `<workflow-id>` with the ID of your Unstructured workflow.
   * Replace `<unstructured-api-key>` with your Unstructured API key value.

4. Click the disk (**Save project to Drive**) icon.

## Step 4: Create the script trigger

1. With the project still open, on the sidebar, click the alarm clock (**Triggers**) icon.

2. Click the **+ Add Trigger** button.

3. Set the following values:

   * For **Choose which function to run**, select `checkForNewOrUpdatedFiles`.

   * For **Choose which deployment should run**, select **Head**.

   * For **Select event source**, select **Time-driven**.

   * For **Select type of time based trigger**, select **Minutes timer**.

   * For **Select minute interval**, select **Every 5 minutes**.

     <Note>
       If you change **Minutes timer** or **Every 5 minutes** to a different interval, you should also go back and change the number `5` in the following
       line of code in the `checkForNewOrUpdatedFiles` function. Change the number `5` to the number of minutes that correspond to the alternate interval you
       selected:

       ```javascript theme={null}
       const thresholdMillis = 5 * 60 * 1000;
       ```
     </Note>

   * For **Failure notification settings**, select an interval such as immediately, hourly, or daily.

4. Click **Save**.

## Step 5: View trigger results

1. With the project still open, on the sidebar, click the three lines (**Executions**) icon.

2. As soon as the first script execution completes, you should see a corresponding message appear in the **Executions** list. If the **Status** column shows
   **Completed**, then keep going with this procedure.

   If the **Status** column shows **Failed**, expand the message to
   get any details about the failure. Fix the failure, and then wait for the next script execution to complete.

3. When the **Status** column shows **Completed** then, in your Unstructured account, click **Jobs** on the sidebar to see if a new job
   is running for that worklow.

   If no new job is running for that workflow, then add at least one new file to—or update at least one existing file in—the Google Drive shared folder or shared drive,
   within 5 minutes of the next script execution. After the next script execution, check the **Jobs** list again.

## Step 6 (Optional): Delete the trigger

1. To stop the script from automatically executing on a regular basis, with the project still open, on the sidebar, click the alarm clock (**Triggers**) icon.
2. Rest your mouse pointer on the trigger you created in Step 4.
3. Click the ellipsis (three dots) icon, and then click **Delete trigger**.


# IBM watsonx Orchestrate
Source: https://docs.unstructured.io/examplecode/tools/ibm-orchestrate



[IBM watsonx Orchestrate](https://www.ibm.com/products/watsonx-orchestrate) helps you build, deploy and manage powerful
AI assistants and agents that automate workflows and processes with generative AI.

This article provides a hands-on, step-by-step walkthrough that uses IBM watsonx Orchestrate to
build a simple AI chat app. This chat app relies on data that is stored in an Astra DB or Milvus vector database. This data
is generated by Unstructured and is based on your organization's source documents and semi-structured data.

## Requirements

To use this example, you will need the following:

If you want to connect Unstructured to Astra DB for providing the source data to IBM watsonx Orchestrate, you will need the following:

* An [IBM Cloud account](https://cloud.ibm.com/registration) or [DataStax account](https://astra.datastax.com/signup) account.

* An [Astra DB database](https://accounts.datastax.com/session-service/v1/login).

* To complete this example, you will need the following settings for the Astra DB database:

  * The database's API endpoint.
  * An application token for the database.
  * The name of the target keyspace in the database.
  * The name of the target collection in the keyspace.

  To get these settings, see the [Astra DB destination connector](/pipelines/destinations/astradb) documentation.

* Within your Unstructured account, a workflow that contains an Astra DB destination connector.

  * Create an [Unstructured account](https://unstructured.io/?modal=try-for-free).
  * Create an [Astra DB destination connector](/pipelines/destinations/astradb) in your Unstructured account.
  * Create a [custom workflow](/pipelines/workflows#create-a-custom-workflow) that contains the Astra DB destination connector in your Unstructured account.

    The workflow must generate [embeddings](/concepts/embedding). The workflow's **Embedder** node's selected embedding model provider must be **IBM**. The
    node's selected embedding model must match the embedding model that you will specify later in **Step 2**.

    Be sure to [run the workflow](/pipelines/workflows#edit%2C-delete%2C-or-run-a-workflow) to have Unstructured generate the data and store it in your Astra DB database.

After you meet the preceding requirements, skip ahead to [Step 1](#step-1%3A-add-an-ibm-watsonx-orchestrate-subscription-to-your-ibm-cloud-account).

If, however, you want to connect Unstructured to Milvus on IBM watsonx.data for providing the source data to IBM watsonx Orchestrate, you will need the following:

* Within your IBM Cloud account, an IBM watsonx.data subscription that contains a Milvus service instance.

  * Create an [IBM Cloud account](https://cloud.ibm.com/registration).

    To complete this example, you must [create an API key](https://www.ibm.com/docs/en/masv-and-l/cd?topic=cli-creating-your-cloud-api-key) for your IBM Cloud account.

  * Create an [IBM watsonx.data subscription](https://cloud.ibm.com/watsonxdata) in your IBM Cloud account.

  * Create a [Milvus service instance](/pipelines/destinations/milvus) within your IBM watsonx.data subscription plan. This instance
    must contain a database, a collection, and an index to store and manage the data that is generated by Unstructured.

    To complete this example, you will need the following settings for the Milvus service instance:

    * The instance's GRPC host value.
    * The instance's GRPC port value.
    * The name of the target database on the instance.
    * The name of the target collection in the database.
    * The name of the target index in the collection.

    To get these settings, see the [Milvus destination connector](/pipelines/destinations/milvus) documentation.

* Within your Unstructured account, a workflow that contains a Milvus destination connector.

  * Create an [Unstructured account](https://unstructured.io/signup).
  * Create a [Milvus destination connector](/pipelines/destinations/milvus) in your Unstructured account.
  * Create a [custom workflow](/pipelines/workflows#create-a-custom-workflow) that contains the Milvus destination connector in your Unstructured account.

    The workflow must generate [embeddings](/concepts/embedding). The workflow's **Embedder** node's selected embedding model provider must be **IBM**. The
    node's selected embedding model must match the embedding model that you will specify later in **Step 2**.

    Be sure to [run the workflow](/pipelines/workflows#edit%2C-delete%2C-or-run-a-workflow) to have Unstructured generate the data and store it in your Milvus instance's database.

## Step 1: Add an IBM watsonx Orchestrate subscription to your IBM Cloud account

If you already have an IBM watsonx Orchestrate subscription, then skip ahead to [Step 2](#step-2%3A-create-the-chat-app-in-ibm-watsonx-orchestrate)

1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
   top navigation bar.
3. Click **Create resource**.
4. With **IBM Cloud catalog** selected, search for and select **watsonx Orchestrate**.
5. Complete the on-screen instructions to finish creating the IBM watsonx Orchestrate subscription.

## Step 2: Create the chat app in IBM watsonx Orchestrate

In this step, you use IBM watsonx Orchestrate to create a chat app. This chat app allows you to ask questions about
your organization's documents and semi-structured data. This data is stored in your Astra DB or Milvus vector database and was
generated by Unstructured in a format that is well-suited for your chat app.

1. Open your IBM watsonx Orchestrate subscription, if it is not already open. To do this:

   a. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).<br />
   b. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
   top navigation bar.<br />
   c. In the list of resources, expand **AI / Machine Learning**, and then click the target watsonx Orchestrate subscription.<br />
   d. Click **Launch watsonx Orchestrate**.<br />

2. If the **Chat** page is not already open, click the **Open the main menu** icon to the far left of the
   top navigation bar, and then click **Chat**.

3. Toward the bottom of the sidebar, click **Create new agent**.

4. On the **Create an agent** page, with **Create from scratch** already selected, enter some **Name** and **Description** for the agent, and then click **Create**.

5. Under **Knowledge source**, in the **Start by adding knowledge** tile, click **Choose knowledge**.

If you are using Astra DB for providing the source data to IBM watsonx Orchestrate, then do the following:

1. On the **Select source** page, click **Astra DB**, and then click **Next**.

2. On the **Connect Astra DB** page, specify the following settings:

   a. For **URL**, enter the API endpoint for the Astra DB database.<br />
   b. Leave **Astra DB port** blank.<br />
   c. For **API key**, enter the application token for your Astra DB database.<br />
   d. Click **Next**.<br />

3. On the **Settings details** page, specify the following settings:

   a. For **Keyspace**, select the name of the target keyspace in the Astra DB database.<br />
   b. For **Data type**, select **Collection**.<br />
   c. For **Collection**, select the name of the target collection in the database.<br />
   d. For **Embedding mode**, select **Client**.<br />
   e. For **Embedding model**, select the name of the embedding model that matches the one that you specified earlier in your Unstructured workflow.<br />
   f. For **Search mode**, select **Vector**.<br />
   g. For **Title**, enter `record_id`.<br />
   h. For **Body**, enter `content`.<br />
   i. Leave **URL** and **Advanced settings** blank.<br />
   j. Click **Next**.<br />

4. On the **Description** page, enter some description for the agent, and then click **Save**.

5. In the list of pages, click **Behavior**.

6. At the bottom of the **Behavior** section, turn on **Chat with documents**.

7. In the upper-right corner of the page, click **Deploy**.

8. On the **Pre-deployment summary** page, click **Deploy**.

Skip ahead to [Step 3](#step-3%3A-run-the-chat-app).

If, however, you are using Milvus on IBM watsonx.data for providing the source data to IBM watsonx Orchestrate, then do the following:

1. On the **Select source** page, click **Milvus**, and then click **Next**.

2. On the **Connect Milvus** page, specify the following settings:

   a. For **GRPC host**, enter the GRPC host value for the Milvus service instance within your IBM watsonx.data subscription.<br />
   b. For **GRPC port**, enter the GRPC port value for the instance.<br />
   c. For **Choose an authentication type**, select **Basic authentication**.<br />
   d. For **Username**, enter `ibmlhapikey`.<br />
   e. For **Password**, enter the API key for your IBM Cloud account.<br />
   f. Click **Next**.<br />

3. On the **Select index** page, specify the following settings:

   a. For **Database**, select the name of the target database on the Milvus service instance within your IBM watsonx.data subscription.<br />
   b. For **Use Collection or Alias**, select **Collection**.<br />
   c. For **Collection**, select the name of the target collection in the database.<br />
   d. For **Index**, select the name of the target index in the collection.<br />
   e. For **Embedding model**, select the name of the embedding model that matches the one that you specified earlier in your Unstructured workflow.<br />
   f. For **Title**, select **element\_id**.<br />
   g. For **Body**, select **text**.<br />
   h. Click **Next**.<br />

4. On the **Description** page, enter some description for the agent, and then click **Save**.

5. At the bottom of the **Behavior** section, turn on **Chat with documents**.

6. In the upper-right corner of the page, click **Deploy**.

7. On the **Pre-deployment summary** page, click **Deploy**.

## Step 3: Run the chat app

In this step, you ask questions about your organization's source documents and semi-structured data. The chat app then
attempts to answer your questions by searching the related data that Unstructured generated and stored in your Astra DB or Milvus vector database.

1. If the **Chat** page is not already open in IBM watsonx Orchestrate, click the **Open the main menu** icon to the far left of the
   top navigation bar, and then click **Chat**.
2. In the sidebar, in the **Agents** list, select the name of the agent that you created in the previous step.
3. In the **Type something** box, enter a question, and then press `Enter`.
4. The agent will provide an answer.
5. Keep asking as many questions as you want to.


# Query JSON with jq
Source: https://docs.unstructured.io/examplecode/tools/jq



[jq](https://jqlang.org/) is a lightweight and flexible command-line JSON processor. You can use `jq` on a local development machine to
slice, filter, map, and transform the JSON data that Unstructured outputs in much the same ways that tools such as `sed`, `awk`, and `grep` let you work with text.

To get `jq`, see the [Download jq](https://jqlang.org/download/) page.

<Info>
  `jq` is not owned or supported by Unstructured. For questions about `jq`and
  feature requests for future versions of `jq`, see the [Issues](https://github.com/jqlang/jq/issues) tab of the
  `jq` repository in GitHub.
</Info>

The following command examples use `jq` with the
[spring-weather.html.json](https://github.com/Unstructured-IO/unstructured/blob/main/example-docs/spring-weather.html.json) file in the
**example-docs** directory within the **Unstructured-IO/unstructured** repository in GitHub.

Find the element with a `type` of `Address`, and print the element's `text` field's value.

```bash theme={null}
jq '.[] 
  | select(.type == "Address") 
  | .text' spring-weather.html.json
```

The output is:

```bash theme={null}
"Silver Spring, MD 20910"
```

Find all elements with a `type` of `Title`, and print the `text` field of each found element as a string in a JSON array.

```bash theme={null}
jq '[
  .[] 
  | select(.type == "Title") 
  | .text]' spring-weather.html.json
```

The output is:

```bash theme={null}
[
  "News Around NOAA",
  "National Program",
  "Are You Weather-Ready for the Spring?",
  "Weather.gov >",
  "News Around NOAA > Are You Weather-Ready for the Spring?",
  "US Dept of Commerce",
  "National Oceanic and Atmospheric Administration",
  "National Weather Service",
  "News Around NOAA",
  "1325 East West Highway",
  "Comments? Questions? Please Contact Us.",
  "Disclaimer",
  "Information Quality",
  "Help",
  "Glossary",
  "Privacy Policy",
  "Freedom of Information Act (FOIA)",
  "About Us",
  "Career Opportunities"
]
```

Find all elements with a `type` of `Title`. Of these, find the ones that have a `text` field that contains the phrase `Contact Us`, and print the contents of each found element's `metadata.link_urls` field.

```bash theme={null}
jq '.[] 
  | select(.type == "Title") 
  | select(.text 
  | contains("Contact Us")) 
  | .metadata.link_urls' spring-weather.html.json
```

The output is:

```bash theme={null}
[
    "https://www.weather.gov/news/contact"
]
```

Find all elements with a `type` of `ListItem`. Of these, find the ones that have a `text` field that contains the phrase `Weather Safety`.
For each item in `metadata.link_texts`, print the item's value as the key, followed by the matching item in
`metadata.link_urls` as the value. Trim any leading and trailing whitespace from all values. Wrap the output in a JSON array.

```bash theme={null}
jq '[
  .[]
  | select(.type == "ListItem")
  | select(.text | test("Weather Safety"; "i"))
  | [.metadata.link_texts, .metadata.link_urls]
  | transpose[]
  | {
      (.[0] | gsub("^\\s+|\\s+$"; "")) : (.[1] | gsub("^\\s+|\\s+$"; ""))
    }
]' spring-weather.html.json
```

The output is:

```bash theme={null}
[
  {
    "Weather Safety": "http://www.weather.gov/safetycampaign"
  },
  {
    "Air Quality": "https://www.weather.gov/safety/airquality"
  },
  {
    "Beach Hazards": "https://www.weather.gov/safety/beachhazards"
  },
  {
    "Cold": "https://www.weather.gov/safety/cold"
  },
  {
    "Cold Water": "https://www.weather.gov/safety/coldwater"
  },
  {
    "Drought": "https://www.weather.gov/safety/drought"
  },
  {
    "Floods": "https://www.weather.gov/safety/flood"
  },
  {
    "Fog": "https://www.weather.gov/safety/fog"
  },
  {
    "Heat": "https://www.weather.gov/safety/heat"
  },
  {
    "Hurricanes": "https://www.weather.gov/safety/hurricane"
  },
  {
    "Lightning Safety": "https://www.weather.gov/safety/lightning"
  },
  {
    "Rip Currents": "https://www.weather.gov/safety/ripcurrent"
  },
  {
    "Safe Boating": "https://www.weather.gov/safety/safeboating"
  },
  {
    "Space Weather": "https://www.weather.gov/safety/space"
  },
  {
    "Sun (Ultraviolet Radiation)": "https://www.weather.gov/safety/heat-uv"
  },
  {
    "Thunderstorms & Tornadoes": "https://www.weather.gov/safety/thunderstorm"
  },
  {
    "Tornado": "https://www.weather.gov/safety/tornado"
  },
  {
    "Tsunami": "https://www.weather.gov/safety/tsunami"
  },
  {
    "Wildfire": "https://www.weather.gov/safety/wildfire"
  },
  {
    "Wind": "https://www.weather.gov/safety/wind"
  },
  {
    "Winter": "https://www.weather.gov/safety/winter"
  }
]
```

## Additional resources

* [jq Tutorial](https://jqlang.org/tutorial/)
* [jq Manual](https://jqlang.org/manual/)
* [jq Playground](https://play.jqlang.org/)


# Working with Unstructured JSON output
Source: https://docs.unstructured.io/examplecode/tools/json-tools



You can use tools such as the following to work with Unstructured JSON output.

## Visual Studio Code

<Note>
  Visual Studio Code is not owned or supported by Unstructured. Unstructured makes no guarantees about the accuracy,
  reliability, or security of this tool or the extensions that you install into it. Use this tool and these
  extensions at your own risk.
</Note>

You can use [Visual Studio Code](https://code.visualstudio.com/) to work with Unstructured JSON output, as follows.

### Format JSON for easier reading in Visual Studio Code

To format an Unstructured JSON output file for easier reading, do the following after you download the Unstructured JSON output file locally and then open the downloaded file in Visual Studio Code:

1. Click anywhere in the JSON file on-screen to make it the active file.

2. Do one of the following:

   * Press **Shift + Option + F** for macOS, or **Shift + Alt + F** for Windows or Linux.
   * On the menu bar, click **View > Command Palette**. Then, in the **Command Palette**, type `>Format Document`, and press **Enter**.

3. The JSON file's contents display in a format that is easier to read.

### Visually render text\_as\_html field contents in Visual Studio Code

To visually render the contents of a `text_as_html` field in Unstructured JSON output, do the following in Visual Studio Code:

1. Install the [HTML Preview](https://marketplace.visualstudio.com/items?itemName=george-alisson.html-preview-vscode)
   extension into your installation of Visual Studio Code, as follows:

   a. On the menu bar, click **View > Extensions**.<br />
   b. In the **Search Extensions in Marketplace** box, type `HTML Preview`, and then press **Enter**.<br />
   c. In the list of search results, for the **HTML Preview** extension, click **Install**.<br />
   d. If prompted, click **Trust Publisher & Install**.<br />

2. Select the contents of the `text_as_html` field that you want to render. Do not copy the
   field name `"text_as_html"` or the opening or closing quotation marks around the field's contents.

3. Create a new file in Visual Studio Code, and paste the contents of the `text_as_html` that you copied, into the new file.

4. With the new file active on-screen, press the key combination `Control + Shift + V` or `Command + Shift + V` to visually render the new file's content.

See also:

* [Table-specific metadata](/concepts/document-elements#table-specific-metadata)
* [VLM generated HTML elements](/concepts/document-elements#vlm-generated-html-elements)
* [Extract tables as HTML](/api-reference/legacy-api/partition/text-as-html) (Unstructured API Reference)
* [Extract tables as HTML](/open-source/how-to/text-as-html) (Unstructured Open Source Reference)

### View image\_base64 field contents as an image in Visual Studio Code

The contents of `image_base64` fields in Unstructured JSON output are in Base64-encoded format. To view the contents of an `image_base64` field as a human-viewable image,
do the following in Visual Studio Code.

1. Install the [Base64Viewer](https://marketplace.visualstudio.com/items?itemName=JasonMejane.base64viewer)
   extension into your installation of Visual Studio Code, as follows:

   a. On the menu bar, click **View > Extensions**.<br />
   b. In the **Search Extensions in Marketplace** box, type `Base64Viewer`, and then press **Enter**.<br />
   c. In the list of search results, for the **Base64Viewer** extension, click **Install**.<br />
   d. If prompted, click **Trust Publisher & Install**.<br />

2. Copy to your system clipboard the contents of the `image_base64` field that you want to decode. Do not copy the
   field name `"image_base64"` or the opening or closing quotation marks around the field's contents.

3. On the menu bar, click **View > Command Palette**.

4. In the **Command Palette**, type `>Base64Viewer: Decode From Base64`, and press **Enter**.

5. When prompted, paste the contents of the `image_base64` field that you copied, and then press **Enter**. The
   human-viewable image appears in a separate editor tab.

See also:

* [Extract images and tables from documents](/api-reference/legacy-api/partition/extract-image-block-types) (Unstructured API Reference)
* [Extract images and tables from documents](/open-source/how-to/extract-image-block-types) (Unstructured Open Source Reference)

### View orig\_elements field contents as original JSON in Visual Studio Code

The contents of `orig_elements` fields in Unstructured JSON output are in gzipped Base64-encoded format. To view the contents of an `orig_elements` field in a human-readable format,
do the following in Visual Studio Code:

1. Install the [Gzip / Unzip text](https://marketplace.visualstudio.com/items?itemName=xballoy.gzip-unzip-text)
   extension into your installation of Visual Studio Code, as follows:

   a. On the menu bar, click **View > Extensions**.<br />
   b. In the **Search Extensions in Marketplace** box, type `Gzip / Unzip text`, and then press **Enter**.<br />
   c. In the list of search results, for the **Gzip / Unzip text** extension, click **Install**.<br />
   d. If prompted, click **Trust Publisher & Install**.<br />

2. Download the Unstructured JSON output file locally, and then open the downloaded file in Visual Studio Code.

   <Note>
     We do not recommend that you format the JSON file for easier reading after you open it in Visual Studio Code.
     This is because the formatting operation could introduce
     unnecessary whitespace and line breaks into the contents of the `orig_elements`
     fields. These additional characters could cause the extension to produce unexpected results or fail.
   </Note>

3. Select the contents of the `orig_elements` field that you want to decode and view. Do not select the
   field name `"orig_elements"` or the opening or closing quotation marks around the field's contents.

4. Right-click the selected contents, and then click **Command Palette**.

5. In the **Command Palette**, type `>Unzip Base64 encoded text`, and then press **Enter**. The human-readable JSON
   appears in a separate editor tab.

See also:

* [Get chunked elements](/api-reference/legacy-api/partition/get-chunked-elements) (Unstructured API Reference)
* [Get chunked elements](/open-source/how-to/get-chunked-elements) (Unstructured Open Source Reference)

## Online web tools

You can use online web tools such as the following to work with Unstructured JSON output.

### Format JSON for easier reading online

<Warning>
  Do not enter any sensitive information into this publicly-available website.
</Warning>

<Note>
  The following website is not owned or supported by Unstructured. Unstructured makes no guarantees about the accuracy,
  reliability, or security of this website. Use this website at your own risk.
</Note>

To format Unstructured JSON output for easier reading, do the following:

1. Go to the [JSON Viewer Online](https://easyjsonviewer.com).

2. Do one of the following:

   * Click **Load**, and then select an Unstructured JSON output file that you have saved locally.
   * Copy the Unstructured JSON output to your system clipboard, and then click **Paste**.

3. Click **Format**. The JSON is formatted on-screen for easier reading.

You should now be able to use your web browser's standard "find" function to search for specific text in
the formatted JSON, as needed.

### Visually render text\_as\_html field contents online

<Warning>
  Do not enter any sensitive information into this publicly-available website.
</Warning>

<Note>
  The following website is not owned or supported by Unstructured. Unstructured makes no guarantees about the accuracy,
  reliability, or security of this website. Use this website at your own risk.
</Note>

To visually render the contents of a `text_as_html` field in Unstructured JSON output, do the following:

1. Go to the [HTML Online Viewer](https://html.onlineviewer.net/).
2. Do one of the following:

   * Copy the contents of the `text_as_html` field to your system clipboard, and then paste it into the **Put your HTML text here** box.
     This content must not contain the field name `"text_as_html"` or the opening or closing quotation marks around the field's contents.
   * Click **Import**, and then select a file containing only the contents of the `text_as_html` field. This file
     must not contain the field name `"text_as_html"` or the opening or closing quotation marks around the field's contents.

<Tip>
  To add a one-pixel, black border to each cell in the rendered HTML for easier reading,
  add the following CSS directive before the opening `<table>` tag:

  ```text theme={null}
  <style>table { border-collapse: collapse; } th, td { border: 1px solid black; }</style> 
  ```

  To add a bit of visual padding to each cell, add the following CSS directive before the
  opening `<table>` tag instead:

  ```text theme={null}
  <style>th, td { border: 1px solid black; padding: 5px; }</style> 
  ```
</Tip>

After you paste or import the `text_as_html` field contents, the HTML is rendered on-screen.

See also:

* [Table-specific metadata](/concepts/document-elements#table-specific-metadata)
* [VLM generated HTML elements](/concepts/document-elements#vlm-generated-html-elements)
* [Extract tables as HTML](/api-reference/legacy-api/partition/text-as-html) (Unstructured API Reference)
* [Extract tables as HTML](/open-source/how-to/text-as-html) (Unstructured Open Source Reference)

### View image\_base64 field contents as an image online

<Warning>
  Do not enter any sensitive information into this publicly-available website.
</Warning>

<Note>
  The following website is not owned or supported by Unstructured. Unstructured makes no guarantees about the accuracy,
  reliability, or security of this website. Use this website at your own risk.
</Note>

The contents of `image_base64` fields in Unstructured JSON output are in Base64-encoded format. To view the contents of an `image_base64` field as a human-viewable image,
do the following:

1. Copy to your system clipboard the contents of the `image_base64` field that you want to view. Do not copy the
   field name `"image_base64"` or the opening or closing quotation marks around the field's contents.
2. Go to the [Base64 to Image](https://base64.guru/converter/decode/image) page on the Base64.Guru website.
3. In the **Base64** box, paste the contents of the `image_base64` field that you copied.
4. Click **Decode Base64 to Image**. The human-viewable image appears on-screen.

See also:

* [Extract images and tables from documents](/api-reference/legacy-api/partition/extract-image-block-types) (Unstructured API Reference)
* [Extract images and tables from documents](/open-source/how-to/extract-image-block-types) (Unstructured Open Source Reference)

### View orig\_elements field contents as original JSON online

<Warning>
  Do not enter any sensitive information into this publicly-available website.
</Warning>

<Note>
  The following website is not owned or supported by Unstructured. Unstructured makes no guarantees about the accuracy,
  reliability, or security of this website. Use this website at your own risk.
</Note>

The contents of `orig_elements` fields in Unstructured JSON output are in gzipped Base64-encoded format. To view the contents of an `orig_elements` field in a human-readable format,
do the following:

1. Go to the [Gzip Decompress Online](https://codebeautify.org/gzip-decompress-online) page on the Code Beautify website.

2. Do one of the following:

   * Copy the contents of the `orig_elements` field to your system clipboard, and then paste it into the **Enter your Gzip Compress Text** box.
     This content must not contain the field name `"orig_elements"` or the opening or closing quotation marks around the field's contents.<br />
   * Click **File**, and then select a file containing only the contents of the `orig_elements` field. This file
     must not contain the field name `"orig_elements"` or the opening or closing quotation marks around the field's contents.<br />

3. If the **Auto** box is not checked, click **Gzip Decompress**.

4. The human-readable JSON appears on-screen.

See also:

* [Get chunked elements](/api-reference/legacy-api/partition/get-chunked-elements) (Unstructured API Reference)
* [Get chunked elements](/open-source/how-to/get-chunked-elements) (Unstructured Open Source Reference)


# Langflow
Source: https://docs.unstructured.io/examplecode/tools/langflow



[Langflow](https://www.langflow.org/) is a visual framework for building multi-agent and RAG applications.
It is open-source, fully customizable, and works with most LLMs and many vector stores out of the box.

<img alt="Langflow designer" />

This no-code, hands-on demonstration walks you through creating a Langflow project that enables you to use GPT-4o-mini to chat
in real time with a PDF document that is processed by Unstructured and has its processed data stored in an
[Astra DB](https://www.datastax.com/products/datastax-astra) vector database.

## Prerequisites

<iframe title="YouTube video player" />

* An IBM Cloud account or DataStax account.

  * For an IBM Cloud account, [sign up](https://cloud.ibm.com/registration) for an IBMid, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your IBMid.
  * For a DataStax account, [sign up](https://astra.datastax.com/signup) for a DataStax account, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your DataStax account.

* An Astra DB database in the DataStax account. To create a database:

  a. After you sign in to DataStax, click **Create database**.<br />
  b. Click the **Serverless (vector)** tile, if it is not already selected.<br />
  c. For **Database name**, enter some unique name for the database.<br />
  d. Select a **Provider** and a **Region**, and then click **Create database**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).

* An application token for the database. To create an application token:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Overview** tab, under **Database Details**, in the **Application Tokens** tile, click **Generate Token**.<br />
  c. Enter some **Token description** and select and **Expiration** time period, and then click **Generate token**.<br />
  d. Save the application token that is displayed to a secure location, and then click **Close**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).

* A keyspace in the database. To create a keyspace:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select **Create keyspace**.<br />
  c. Enter some **Keyspace name**, and then click **Add keyspace**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-keyspaces.html#keyspaces).

* A collection in the keyspace.

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * An existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. If Unstructured generates embeddings,
      the new collection's name will be `u<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `u<short-workflow-id`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * For the source connector only, an existing collection is required.
  * For the destination connector only, an existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. The new collection's name will be `unstructuredautocreated`.

  To create a collection yourself:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select the name of the target keyspace.<br />
  c. In the **Collections** list, select **Create collection**.<br />
  d. Enter some **Collection name**.<br />
  e. Turn on **Vector-enabled collection**, if it is not already turned on.<br />
  f. Choose a mode for **Embedding generation method**. See [Astra DB generated embeddings](#astra-db-generated-embeddings).<br />
  g. If you chose **Bring my own**, enter the number of dimensions for the embedding model that you plan to use.<br />
  h. For **Similarity metric**, select **Cosine**.<br />
  i. Click **Create collection**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

Also:

* [Sign up for an OpenAI account](https://platform.openai.com/signup), and [get your OpenAI API key](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key).
* [Sign up for a free Langflow account](https://astra.datastax.com/signup?type=langflow).
* Get your Unstructured account and Unstructured API key:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

## Create and run the demonstration project

<Steps>
  <Step title="Create the Langflow project">
    1. Sign in to your Langflow dashboard.
    2. From your dashboard, click **New Project**.
    3. Click **Blank Flow**.
  </Step>

  <Step title="Add the Unstructured component">
    In this step, you add a component that instructs the legacy Unstructured Partition Endpoint to process a local file that you specify.

    1. On the sidebar, expand **Experimental (Beta)**, and then expand **Loaders**.

    2. Drag the **Unstructured** component onto the designer area.

    3. In the **Unstructured** component, click the box or icon next to **File**, and then select a local file for Unstructured to process.

       This component works only with the file extensions `.pdf`, `.docx`, and `.txt`. Although you can use any local file with one of these extensions,
       this demonstration uses [the text of the United States Constitution in PDF format](https://constitutioncenter.org/media/files/constitution.pdf),
       saved to your local development machine.

       <Tip>
         To work with multiple local files, or to work with remote files, see the suggested options in [Next steps](#next-steps).
       </Tip>

    4. For **Unstructured.io Serverless API Key**, enter your Unstructured API key value.

       <img alt="Unstructured component" />

    5. Wait until **Saved** appears in the top navigation bar.

       <img alt="Saved message" />
  </Step>

  <Step title="Add the OpenAI Embeddings component">
    In this step, you add a component that generates vector embeddings for the processed data that Unstructured outputs.

    1. On the sidebar, expand **Embeddings**, and then drag the **OpenAI Embeddings** component onto the designer area.

    2. In the **OpenAI Embeddings** component, for **Model**, select `text-embedding-3-large`.

    3. For **OpenAI API Key**, enter your OpenAI API key's value.

       <img alt="OpenAI Embeddings component" />

    4. Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Astra DB components">
    In this step, you add two components. The first component instructs Astra DB to ingest into the specified Astra DB collection the processed data that Unstructured outputs along
    with the associated generated vector embeddings. The second component instructs Astra DB to take user-supplied chat messages and perform contextual
    searches over the ingested data in the specified Astra DB collection, outputting its search results.

    1. On the sidebar, expand **Vector Stores**, and then drag the **Astra DB** component onto the designer area.

    2. Double-click the **Astra DB** component's title bar, and rename the component to `Astra DB Ingest`.

    3. Repeat these previous two actions to add a second **Astra DB** component, renaming it to `Astra DB RAG`.

    4. In both of these **Astra DB** components, in the **Database** list, select the name of your Astra DB database. Make sure this is the same database name in both components.

    5. In the **Collection** list in both components, select the name of the collection in the database. Make sure this is the same collection name in both components.

    6. In the **Astra DB Application Token** box in both components, enter your Astra DB application token's value. Make sure this is the same application token value in both components.

    7. Connect the **Data** output from the **Unstructured** component to the **Ingest Data** input in the **Astra DB Ingest** component.

       To make the connection, click and hold your mouse pointer inside of the circle next to **Data** in the **Unstructured** component.
       While holding your mouse pointer, drag it over into the circle next to **Ingest Data** in the **Astra DB Ingest** component. Then
       release your mouse pointer. A line appears between these two circles.

    8. Connect the **Embeddings** output from the **OpenAI Embeddings** component to the **Embedding or Astra Vectorize** input in the  **Astra DB Ingest** component.

       <img alt="Astra DB component" />

    9. Wait until **Saved** appears in the top navigation bar.

    10. In the title bar of the **Astra DB Ingest** component, click the play icon. This ingests the processed data
        from Unstructured and the associated generated vector embeddings into the specified Astra DB collection.

        <img alt="Play icon" />

    11. Wait until **Building** disppears from the top navigation bar, and a green check mark appears next to this play icon. This could take several minutes.

        <Note>
          Each time you click the play icon in the **Astra DB Ingest** component, Unstructured reprocesses the specified local
          file. If this file does not change, this could result in multiple duplicate records
          being inserted into the specified Astra DB collection. You should only click the play icon in the **Astra DB Ingest** component when you want to insert new processed data into
          the specified Astra DB collection.
        </Note>
  </Step>

  <Step title="Add the Chat Input component">
    In this step, you add a component that takes user-supplied chat messages and sends them as input to Astra DB for contextual searching.

    1. On the sidebar, expand **Inputs**, and then drag the **Chat Input** component onto the designer area.

    2. Connect the **Message** output from the **Chat Input** component to the **Search Input** input in the **Astra DB RAG** component.

       <img alt="Chat Input component" />

    3. Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Parse Data component">
    In this step, you add a component that takes the Astra DB search results and converts them into plain text, suitable for inclusion in
    a prompt to a text-based LLM.

    1. On the sidebar, expand **Helpers**, and then drag the **Parse Data** component onto the designer area.

    2. Connect the **Search Results** output from the **Astra DB RAG** component to the **Data** input in the **Parse Data** component.

       <img alt="Parse Data component" />

    3. Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Prompt component">
    In this step, you add a component that builds a prompt and then sends it to a text-based LLM.

    1. On the sidebar, expand **Prompts**, and then drag the **Prompt** component onto the designer area.

    2. In the **Prompt** component, next to **Template**, click the box or arrow icon.

    3. In the **Edit Prompt** window, enter the following prompt:

       ```text theme={null}
       {context}

       ---

       Given the context above, answer the question as best as possible.

       Question: {question}

       Answer: 
       ```

       <Tip>
         To answer the question, the preceding prompt uses the context along with general information that the text-based LLM is
         trained on. To use only the context to answer the question, you can change the prompt, for example to something like this:

         ```text theme={null}
         {context}

         ---

         Given the context above, answer the question as best as possible. Use only the context to answer the question. Do not use 
         any other sources of information. If the context does not provide enough information to answer the question, reply with 
         'I do not have enough context to answer the question.'

         Question: {question}

         Answer:
         ```
       </Tip>

    4. Click **Check & Save**.

       <img alt="Prompt component" />

    5. Connect the **Text** output from the **Parse Data** component to the **context** input in the **Prompt** component.

       <img alt="Connect Prompt component" />

    6. Connect the **Message** output from the **Chat Input** component to the **question** input in the **Prompt** component.

       <Note>
         You will now have two connections from the **Message** output in the **Chat Input** component:

         * One connection was already made to the **Search Input** input in the **Astra DB RAG** component.
         * Another connection has just now been made to the **question** input in the **Prompt** component.
       </Note>

    7. Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the OpenAI component">
    In this step, you create a component that sends a prompt to a text-based LLM and outputs the LLM's response.

    1. On the sidebar, expand **Models**, and then drag the **OpenAI** component onto the designer area.

    2. In the **Model Name** list, select **gpt-4o-mini**.

    3. For **OpenAI API Key**, enter your OpenAI API key's value.

    4. For **Temperature**, enter `0.1`.

    5. Connect the **Prompt Message** output from the **Prompt** component to the **Input** input in the **OpenAI** component.

       <img alt="OpenAI component" />

    6. Wait until **Saved** appears in the top navigation bar.
  </Step>

  <Step title="Add the Chat Output component">
    In this step, you create a component that returns the answer to the user's original chat message.

    1. On the sidebar, expand **Outputs**, and then drag the **Chat Output** component onto the designer area.

    2. Connect the **Text** output from the **OpenAI** component to the **Text** input in the **Chat Output** component.

       <img alt="Chat Output component" />

    3. Wait until **Saved** appears in the top navigation bar.

    The final project should look like this:

    <img alt="Final project results" />
  </Step>

  <Step title="Run the project">
    1. In the designer area, click **Playground**.

       <img alt="Open Playground button" />

    2. Enter a question into the chat box, for example, `What rights does the fifth amendment guarantee?` Then press the send button.

       <img alt="Playground window" />

    3. Wait until the answer appears.

    4. Ask as many additional questions as you want to.
  </Step>
</Steps>

## Next steps

Now that you have your pipeline set up, here are just a few ways that you could modify it to support different requirements,
such as processing multiple files or using a different vector store.

### Process multiple files

In this demonstration, you pass to Unstructured a single local file. To pass multiple local or
non-local files to Unstructured instead, you can use the
[Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview) or
[Unstructured Ingest](/open-source/ingestion/overview) outside of Langflow.

To do this, you can:

* [Use Unstructured Pipelines to create a workflow](/pipelines/quickstart) that relies on any available
  [source connector](/pipelines/sources/overview) to connect to
  [Astra DB](/pipelines/destinations/astradb). Run this workflow outside of Langflow anytime you have new documents in that source location that
  you want Unstructured to process and then insert the new processed data into Astra DB. Then, back in the Langflow project,
  use the **Playground** to ask additional questions, which will now include the new data when generating answers.

* [Use Unstructured Ingest to create a pipeline](/open-source/ingestion/overview) that relies on any available
  [source connector](/open-source/ingestion/source-connectors/overview) to connect to
  [Astra DB](/open-source/ingestion/destination-connectors/astradb). Run this pipeline outside of Langflow anytime you have new documents in that non-local source location that
  you want Unstructured to process and then insert the new processed data into Astra DB. Then, back in the Langflow project,
  use the **Playground** to ask additonal questions, which will now include the new data when generating answers.

### Use a different vector store

In this demonstration, you use Astra DB as the vector store. Langflow and Unstructured support several vector stores in addition to Astra DB.

To do this, you can:

[Use Unstructured Pipelines to create a workflow](/pipelines/quickstart) that relies on any available
[source connector](/pipelines/sources/overview) to connect to
one of the following available vector stores that Langflow also supports:

* [Milvus](/pipelines/destinations/milvus)
* [MongoDB](/pipelines/destinations/mongodb)
* [Pinecone](/pipelines/destinations/pinecone)

Run this workflow outside of Langflow anytime you have new documents in the source location that
you want Unstructured to process and then insert the new processed data into the vector store. Then, back in the Langflow project,
swap out the **Astra DB RAG** component for the corresponding **Vector Stores** component that matches the new vector
store's name. Configure the new component, and then
use the **Playground** to ask additional questions, which will now use the new vector store when generating answers.

Or, [use Unstructured Ingest to create a pipeline](/open-source/ingestion/overview) that relies on any available
[source connector](/open-source/ingestion/source-connectors/overview) to connect to
one of the following available vector stores that Langflow also supports:

* [Chroma DB](/open-source/ingestion/destination-connectors/chroma)
* [Couchbase](/open-source/ingestion/destination-connectors/couchbase)
* [Elasticsearch](/open-source/ingestion/destination-connectors/elasticsearch)
* [Milvus](/open-source/ingestion/destination-connectors/milvus)
* [MongoDB](/open-source/ingestion/destination-connectors/mongodb)
* [OpenSearch](/open-source/ingestion/destination-connectors/opensearch)
* [Pinecone](/open-source/ingestion/destination-connectors/pinecone)
* [Qdrant](/open-source/ingestion/destination-connectors/qdrant)
* [Vectara](/open-source/ingestion/destination-connectors/vectara)
* [Weaviate](/open-source/ingestion/destination-connectors/weaviate)

Run this pipeline outside of Langflow anytime you have new documents in the source location that
you want Unstructured to process and then insert the new processed data into the vector store. Then, back in the Langflow project,
swap out the **Astra DB RAG** component for the corresponding **Vector Stores** component that matches the new vector
store's name. Configure the new component, and then
use the **Playground** to ask additional questions, which will now use the new vector store when generating answers.

## Learn more

* See the [Langflow documentation](https://docs.langflow.org/).
* <Icon icon="blog" />  [No-Code AI Assistant in No Time with Unstructured Platform, AstraDB, and Langflow](https://unstructured.io/blog/no-code-ai-assistant-in-no-time-with-unstructured-platform-astradb-and-langflow)


# Model Context Protocol (MCP) Hands-On Walkthrough for the Unstructured API's workflow operations
Source: https://docs.unstructured.io/examplecode/tools/mcp



[Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open protocol that standardizes
how applications provide context to LLMs. From the MCP documentation:

> Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.

This article provides a hands-on, step-by-step walkthrough to build a simple example on your local development machine
that integrates the [Unstructured API](/api-reference/overview) with the
[Python SDK for Model Context Protocol](https://github.com/modelcontextprotocol/python-sdk). Specifically,
this walkthrough demonstrates how to write and run Python code to create an MCP *server*
(a lightweight program that exposes specific capabilities through MCP); this MCP server uses the
[Unstructured API's workflow operations](/api-reference/workflow/overview) to list all of your available
[source connectors](/api-reference/workflow/sources/overview) and to get information about a specific source connector. The
Unstructured API's workflow operations offer a full range of partitioning, chunking, embedding, and enrichment options for your files and data.

<Note>
  For a hands-on walkthrough of the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) instead,
  see the [MCP Hands-On Walkthrough for the legacy Unstructured Partition Endpoint](/examplecode/tools/mcp-partition), which
  is intended for rapid prototyping of Unstructured's various partitioning strategies, with limited support for chunking,
  processing local files one file at a time.
</Note>

You call this MCP server from an MCP *client* (an application that maintains a one-to-one connection with an
MCP server), which in this walkthrough is [Claude for Desktop](https://claude.ai/download), a chat-based interface, by asking it plain language questions.
Other MCP clients are also available. [Learn more](https://modelcontextprotocol.io/clients).

At the end of this walkthrough, you can extend this example by using the Unstructured API's workflow operations to perform other tasks, such as creating, listing, changing, and
deleting various connectors, workflows, and jobs in your Unstructured account.

## Requirements

To complete this walkthrough, you must first have an Unstructured account, an Unstructured API key for that account,
the necessary Python toolchain and project setups, and Claude for Desktop, as follows.

### Unstructured account and API key

Before you begin, you must have an Unstructured account and an Unstructured API key, as follows:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

### Python toolchain and Claude for Desktop setup

Before you can start coding on your local machine, you must install the Python package and project manager [uv](https://docs.astral.sh/uv/),
and install Claude for Desktop, as follows.

<Steps>
  <Step>
    Install `uv`, as follows.

    By using `pip`:

    ```bash theme={null}
    pip install uv
    ```

    Or, by using the following script (be sure to restart your terminal or Command Prompt after running the script):

    ```bash theme={null}
    # For macOS and Linux:
    curl -LsSf https://astral.sh/uv/install.sh | sh

    # For Windows:
    powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
    ```

    (To learn more about `uv`, see the [uv documentation](https://docs.astral.sh/uv/).)
  </Step>

  <Step>
    [Install Claude for Desktop](https://claude.ai/download).
  </Step>
</Steps>

You are now ready to start coding.

## Step 1: Create the MCP server

<Tip>
  If you are not interested in building an MCP server from the beginning and want to use a pre-built MCP server instead,
  skip ahead to [Optional Step 4: Extend the MCP server](#optional-step-4%3A-extend-the-mcp-server).
</Tip>

<Steps>
  <Step>
    From your current working directory, create a project directory and then initialize it. This example creates a
    project directory named `mcp-unstructured-demo`. Then go to the project directory:

    ```bash theme={null}
    uv init mcp-unstructured-demo
    cd mcp-unstructured-demo
    ```
  </Step>

  <Step>
    Create a virtual environment in the root of the project directory. Then activate the virtual environment:

    ```bash theme={null}
    # For macOS and Linux:
    uv venv
    source .venv/bin/activate

    # For Windows:
    uv venv
    .venv\Scripts\activate
    ```

    To deactivate and exit the virtual environment at any time, simply run the `deactivate` command:

    ```bash theme={null}
    deactivate
    ```
  </Step>

  <Step>
    With the virtual environment activated, install the Python SDK for Model Context Protocol and additional code dependencies:

    ```bash theme={null}
    uv add "mcp[cli]" "unstructured-client>=0.30.6" dotenv
    ```
  </Step>

  <Step>
    In the root of the project directory, create a new Python file. This example names the file `workflow.py`.
    Then add the following code to this new `workflow.py` file:

    ```python theme={null}
    from typing import Any, AsyncIterator, Optional
    from mcp.server.fastmcp import FastMCP, Context
    from dotenv import load_dotenv
    import os
    from dataclasses import dataclass
    from contextlib import asynccontextmanager
    from unstructured_client import UnstructuredClient
    from unstructured_client.models.operations import (
        ListSourcesRequest,
        GetSourceRequest
    )

    def load_environment_variables() -> None:
        """
        Loads environment variables from a .env file and checks for required variables. 
        Raises an error if any required variable is missing.

        Args:
            None

        Returns:
            None
        """
        load_dotenv()
        required_vars = [
            "UNSTRUCTURED_API_KEY"
        ]

        for var in required_vars:
            if not os.getenv(var):
                raise ValueError(f"Missing required environment variable: {var}")

    @dataclass
    class AppContext:
        client: UnstructuredClient

    @asynccontextmanager
    async def app_lifespan(server: FastMCP) -> AsyncIterator[AppContext]:
        """
        Manages application lifecycle with type-safe context
        for the FastMCP server. Initializes the UnstructuredClient.
        
        Args:
            server (FastMCP): The FastMCP server instance.
        
        Returns:
            AsyncIterator[AppContext]: An asynchronous context manager providing the 
            application context.
        """
        api_key = os.getenv("UNSTRUCTURED_API_KEY")
        if not api_key:
            raise ValueError("UNSTRUCTURED_API_KEY environment variable is required")
        client = UnstructuredClient(api_key_auth=api_key)
        try:
            yield AppContext(client=client)
        finally:
            pass

    # Initialize the FastMCP server.
    mcp = FastMCP("workflow", lifespan=app_lifespan)

    @mcp.tool()
    async def list_sources(ctx: Context, source_type: Optional[str] = None) -> str:
        """
        Lists available source connectors. Optionally filters by source type.
        
        Args:
            ctx (Context): The context for the request.
            source_type (Optional[str]): The type of source connector to filter by. 
                If None, all sources are listed.
        
        Returns:
            str: A formatted string listing available source connector names and IDs.
        """
        client = ctx.request_context.lifespan_context.client
        request = ListSourcesRequest()

        if source_type:
            try:
                request.source_type = [source_type]
            except KeyError:
                return f"Invalid source type: {source_type}"

        response = await client.sources.list_sources_async(
            request=request
        )
        sorted_sources = sorted(
            response.response_list_sources, 
            key=lambda source: source.name.lower()
        )
        result = ["Available source connectors:"]

        for source in sorted_sources:
            result.append(f"- {source.name} (ID: {source.id})")

        return "\n".join(result)

    @mcp.tool()
    async def get_source(ctx: Context, source_id: str) -> str:
        """
        Gets detailed information about a specific source connector by its ID.
        
        Args:
            ctx (Context): The context for the request.
            source_id (str): The ID of the source connector to retrieve.
            
        Returns: 
            str: A formatted string containing information about the source connector.
        """
        client = ctx.request_context.lifespan_context.client
        
        response = await client.sources.get_source_async(
            request=GetSourceRequest(source_id=source_id)
        )
        info = response.source_connector_information

        result = ["Source connector information:"]
        result.append(f"- name: {info.name}")
        
        for key, value in info.config:
            result.append(f"{key}: {value}")

        return "\n".join(result)
        
    if __name__ == "__main__":
        load_environment_variables()
        # Initialize and run the MCP server.
        mcp.run(transport='stdio')
    ```
  </Step>

  <Step>
    In the root of the project directory, create a new file named `.env`.
    Then add the following code to this new `.env` file, replacing `<your-unstructured-api-key>` with your Unstructured API key:

    ```text theme={null}
    UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
    ```
  </Step>
</Steps>

## Step 2: Configure Claude for Desktop to use the MCP server

<Steps>
  <Step>
    Open the Claude for Desktop App configuration file from the following location, for example by using Visual Studio Code:

    ```text theme={null}
    # For macOS or Linux:
    code ~/Library/Application\ Support/Claude/claude_desktop_config.json

    # For Windows:
    code $env:AppData\Claude\claude_desktop_config.json
    ```
  </Step>

  <Step>
    Add the following code to the `claude_desktop_config.json` file, and then save the file.

    In this file:

    * `"workflow"` must match the value that you set in `mcp = FastMCP("workflow" ...` in the `workflow.py` file.
    * "Absolute path to" `uv` must be the absolute path to the `uv` executable on your local machine.
    * "Absolute path to project directory" must be the absolute path to the project directory.
    * `mcp-unstructured-demo` must match the name of the project directory that you created.
    * `"workflow.py"` must match the name of the Python file that you created for the MCP server.

    For macOS or Linux:

    ```json theme={null}
    {
        "mcpServers": {
            "workflow": {
                "command": "/ABSOLUTE/PATH/TO/uv",
                "args": [
                    "--directory",
                    "/ABSOLUTE/PATH/TO/PROJECT/DIRECTORY/mcp-unstructured-demo/",
                    "run",
                    "workflow.py"
                ]
            }
        }
    }
    ```

    For Windows:

    ```json theme={null}
    {
        "mcpServers": {
            "workflow": {
                "command": "C:\\ABSOLUTE\\PATH\\TO\\uv",
                "args": [
                    "--directory",
                    "C:\\ABSOLUTE\\PATH\\TO\\PROJECT\\DIRECTORY\\mcp-unstructured-demo\\",
                    "run",
                    "workflow.py"
                ]
            }
        }
    }
    ```
  </Step>
</Steps>

## Step 3: Run the MCP server and call it from Claude for Desktop

<Steps>
  <Step>
    Start Claude for Desktop. If it is already running, restart it.
  </Step>

  <Step>
    Ask the following question in the chat window, replacing `S3` with the type of source connector that you want to list.
    For a list of available source connector types, see the [Unstructured API's workflow operations' sources overview](/api-reference/workflow/sources/overview).

    ```text theme={null}
    List all available source connectors of type S3.
    ```

    Claude for Desktop responds with a list of all available source connectors for that source connector type.
  </Step>

  <Step>
    Ask another question in the chat window, replacing `<source-connector-id>` with one of the source connector IDs that
    Claude for Desktop returned in the previous step. For example:

    ```text theme={null}
    Provide information about the source connector with ID <source-connector-id>.
    ```

    Claude for Desktop responds with information about that source connector.
  </Step>
</Steps>

## Optional Step 4: Extend the MCP server

Unstructured offers an initial experimental MCP server implementation, with much more functionality then the MCP server that you just created previously in this walkthrough.
This initial MCP server is intended only for experimentation purposes and is not recommended for production usage. A more complete, supported,
production-level implementation is coming soon.

To use Unstructured's initial experimental MCP server implementation, do the following:

<Steps>
  <Step>
    From a different working directory than the one you used in previously in this walkthrough, use Git to clone the
    [Unstructured API MCP Server](https://github.com/Unstructured-IO/UNS-MCP) repository in GitHub, and then switch to the cloned
    repository's root directory:

    ```bash theme={null}
    git clone https://github.com/Unstructured-IO/UNS-MCP.git
    cd UNS-MCP
    ```
  </Step>

  <Step>
    In the cloned repository's root directory, create a new file named `.env`. Then copy the contents of the repository's
    `.env.template` file into the `.env` file that you just created.

    As an MCP security best practice, sensitive information such as API keys should be stored in environment variables or a
    secrets manager, not hard-coded in MCP server code or retrieved from the user by an MCP client.
    The `.env` file is a common way to store environment variables in Python projects outside of MCP server code and not
    visible to MCP clients.

    Having your own `.env` file helps ensure that you do
    not accidentally commit any sensitive information from this cloned repository into other repositories (as `.env` is an
    entry in this cloned repository's `.gitignore` file, but `.env.template` is not).

    In your `.env` file, replace `<key-here>` in the following line with your Unstructured API key:

    ```text theme={null}
    UNSTRUCTURED_API_KEY="<key-here>"
    ```

    As needed, replace any of the other placeholders in the `.env` file with your own values. For example, if you want to
    create an Amazon S3 source or destination connector, you should provide your AWS credentials in the `.env` file by specifying values
    for the `AWS_KEY` and `AWS_SECRET` variables. For more information, see the usage comments in the original `.env.template`
    file or in the content that you copied into your `.env` file.

    The original `.env.template` file will be updated as more features are added to the repository. You should periodically
    pull down the source repository's contents into your cloned repository to get any new features, as well as the latest version
    of the `.env.template` file that you can copy any newly available required environment variable names over into your existing `.env` file.
  </Step>

  <Step>
    In the cloned repository's root directory, create a virtual environment. Then activate the virtual environment:

    ```bash theme={null}
    # For macOS and Linux:
    uv venv
    source .venv/bin/activate

    # For Windows:
    uv venv
    .venv\Scripts\activate
    ```

    To deactivate and exit the virtual environment at any time, simply run the `deactivate` command:

    ```bash theme={null}
    deactivate
    ```
  </Step>

  <Step>
    With the virtual environment activated, install the cloned repository's code dependencies:

    ```bash theme={null}
    uv sync
    ```
  </Step>

  <Step>
    Open the Claude for Desktop App configuration file from the following location, for example by using Visual Studio Code:

    ```text theme={null}
    # For macOS or Linux:
    code ~/Library/Application\ Support/Claude/claude_desktop_config.json

    # For Windows:
    code $env:AppData\Claude\claude_desktop_config.json
    ```
  </Step>

  <Step>
    Overwrite the contents of the `claude_desktop_config.json` file with the following content, and then save the file.

    For macOS or Linux:

    ```json theme={null}
    {
        "mcpServers": {
            "UNS-MCP": {
                "command": "/ABSOLUTE/PATH/TO/uv",
                "args": [
                    "--directory",
                    "/ABSOLUTE/PATH/TO/PARENT/FOLDER/UNS-MCP",
                    "run",
                    "server.py"
                ]
            }
        }
    }
    ```

    For Windows:

    ```json theme={null}
    {
        "mcpServers": {
            "workflow": {
                "command": "C:\\ABSOLUTE\\PATH\\TO\\uv",
                "args": [
                    "--directory",
                    "C:\\ABSOLUTE\\PATH\\TO\\PARENT\\FOLDER\\UNS-MCP",
                    "run",
                    "server.py"
                ]
            }
        }
    }
    ```
  </Step>

  <Step>
    Start Claude for Desktop. If it is already running, restart it.
  </Step>

  <Step>
    Here are some questions that you can ask in the chat window to test the MCP server. Replace the placeholders with the appropriate values.

    For a list of available source connector types, see the [Unstructured API's workflow operations sources overview](/api-reference/workflow/sources/overview).

    For a list of available destination connector types, see the [Unstructured API's workflow operations destinations overview](/api-reference/workflow/destinations/overview).

    * `List all available source connectors of type <source-connector-type>.`
    * `Provide information about the source connector with ID <source-connector-id>.`
    * `List all available source connectors of type <destination-connector-type>.`
    * `Provide information about the source connector with ID <destination-connector-id>.`
    * `List all available destination connectors of type <destination-connector-type>.`
    * `Provide information about the destination connector with ID <destination-connector-id>.`
    * `Create a new workflow.`
    * `Provide information about the workflow with ID <workflow-id>.`
    * `Run the workflow with ID <workflow-id>.`
    * `Update the workflow with ID <workflow-id>.`
    * `List all available workflows.`
    * `Delete the worklow with ID <workflow-id>.`
  </Step>

  <Step>
    New features are being added to the source repository over time, which will increase the types of questions that
    you can ask the MCP server. You should periodically
    pull down the source repository's contents into your cloned repository to get any new features as they are added.
  </Step>
</Steps>

## Learn more

* <Icon icon="blog" />  [Building an MCP Server with Unstructured API](https://unstructured.io/blog/building-an-mcp-server-with-unstructured-api)
* <Icon icon="blog" />  [Building an End-to-End Data Pipeline with Custom NER on Unstructured using MCP](https://unstructured.io/blog/building-an-end-to-end-data-pipeline-with-custom-ner-on-unstructured-using-mcp)
* <Icon icon="blog" />  Uses this MCP server: [All Your Unstructured Data in a Databricks Delta Table. Just Say the Word.](https://unstructured.io/blog/all-your-unstructured-data-in-a-databricks-delta-table-just-say-the-word)


# Model Context Protocol (MCP) Hands-On Walkthrough for the Unstructured Partition Endpoint
Source: https://docs.unstructured.io/examplecode/tools/mcp-partition



[Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open protocol that standardizes
how applications provide context to LLMs. From the MCP documentation:

> Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.

This article provides a hands-on, step-by-step walkthrough to build a simple example on your local development machine
that integrates the [Unstructured API](/api-reference/overview) with MCP. Specifically,
this walkthrough demonstrates how to run an MCP *server*
(a lightweight program that exposes specific capabilities through MCP); this MCP server uses the
[Unstructured Partition Endpoint](/api-reference/workflow/overview) to partition a single file, with limited support
for chunking as well, processing local files one file at a time.

<Note>
  For a hands-on walkthrough of the [Unstructured API's workflow operations](/api-reference/workflow/overview) instead,
  see the [MCP Hands-On Walkthrough for the Unstructured API's workflow operations](/examplecode/tools/mcp), which offers
  a full range of partitioning, chunking, embedding, and enrichment options for your files and data.
</Note>

You call this MCP server from an MCP *client* (an application that maintains a one-to-one connection with an
MCP server), which in this walkthrough is [Claude for Desktop](https://claude.ai/download), a chat-based interface, by asking it plain language questions.
Other MCP clients are also available. [Learn more](https://modelcontextprotocol.io/clients).

## Requirements

To complete this walkthrough, you must first have an Unstructured account, an Unstructured API key for that account,
the necessary Python toolchain and project setups, Claude for Desktop, and an input file for Unstructured to process, as follows.

### Unstructured account and API key

Before you begin, you must have an Unstructured account and an Unstructured API key, as follows:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

### Python toolchain and Claude for Desktop setup

Before you can start coding on your local machine, you must install the Python package and project manager [uv](https://docs.astral.sh/uv/),
and install Claude for Desktop, as follows.

<Steps>
  <Step>
    Install `uv`, as follows.

    By using `pip`:

    ```bash theme={null}
    pip install uv
    ```

    Or, by using the following script (be sure to restart your terminal or Command Prompt after running the script):

    ```bash theme={null}
    # For macOS and Linux:
    curl -LsSf https://astral.sh/uv/install.sh | sh

    # For Windows:
    powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
    ```

    (To learn more about `uv`, see the [uv documentation](https://docs.astral.sh/uv/).)
  </Step>

  <Step>
    [Install Claude for Desktop](https://claude.ai/download).
  </Step>
</Steps>

### Input file

You must have an input file for Unstructured to process. This file can be any file type that Unstructured supports, such as a `.pdf` file.
[See the list of supported file types](/pipelines/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

To minimize the risk of Claude for Desktop's hard timeout limit of 60 seconds for MCP server responses, we recommend that you use small files
(less than 400 KB) for this walkthrough.

You are now ready to start coding.

## Step 1: Create the MCP server

<Steps>
  <Step>
    From your current working directory, create a project directory and then initialize it. This example creates a
    project directory named `mcp-unstructured-parition-demo`. Then go to the project directory:

    ```bash theme={null}
    uv init mcp-unstructured-partition-demo
    cd mcp-unstructured-partition-demo
    ```
  </Step>

  <Step>
    Create a virtual environment in the root of the project directory. Then activate the virtual environment:

    ```bash theme={null}
    # For macOS and Linux:
    uv venv
    source .venv/bin/activate

    # For Windows:
    uv venv
    .venv\Scripts\activate
    ```

    To deactivate and exit the virtual environment at any time, simply run the `deactivate` command:

    ```bash theme={null}
    deactivate
    ```
  </Step>

  <Step>
    With the virtual environment activated, install the Python SDK for Model Context Protocol and additional code dependencies:

    ```bash theme={null}
    uv add "mcp[cli]" "unstructured-client>=0.30.6" dotenv
    ```
  </Step>

  <Step>
    In the root of the project directory, create a new Python file. This example names the file `doc_processor.py`.
    Then add the following code to this new `doc_processor.py` file.

    In the following code, replace `<absolute-path-to-local-directory>` with the absolute path to the local directory where you want
    Unstructured to store the processed files' content. Be sure that this local directory exists before you run the code.

    ```python theme={null}
    import os
    from dotenv import load_dotenv
    import json
    from unstructured_client import UnstructuredClient
    from typing import AsyncIterator
    from dataclasses import dataclass
    from contextlib import asynccontextmanager
    from mcp.server.fastmcp import FastMCP, Context
    from unstructured_client.models import operations, shared

    @dataclass
    class AppContext:
        client: UnstructuredClient

    @asynccontextmanager
    async def app_lifespan(server: FastMCP) -> AsyncIterator[AppContext]:
        """Manages the Unstructured API client lifecycle."""
        api_key = os.getenv("UNSTRUCTURED_API_KEY")
        if not api_key:
            raise ValueError("UNSTRUCTURED_API_KEY environment variable is required")

        client = UnstructuredClient(api_key_auth=api_key)
        try:
            yield AppContext(client=client)
        finally:
            # No cleanup needed for the API client.
            pass

    # Create the MCP server instance.
    mcp = FastMCP("unstructured-partition-mcp", lifespan=app_lifespan, dependencies=["unstructured-client", "python-dotenv"])
    # Specify the absolute path to the local directory to store processed files.
    PROCESSED_FILES_FOLDER = "<absolute-path-to-local-directory>"

    def load_environment_variables() -> None:
        """
        Load environment variables from .env file.
        Raises an error if critical environment variables are missing.
        """
        load_dotenv()
        required_vars = [
            "UNSTRUCTURED_API_KEY"
        ]

        for var in required_vars:
            if not os.getenv(var):
                raise ValueError(f"Missing required environment variable: {var}")

    def json_to_text(file_path) -> str:
        with open(file_path, 'r') as file:
            elements = json.load(file)

        doc_texts = []

        for element in elements:
            text = element.get("text", "").strip()
            element_type = element.get("type", "")
            metadata = element.get("metadata", {})

            if element_type == "Title":
                doc_texts.append(f"<h1> {text}</h1><br>")
            elif element_type == "Header":
                doc_texts.append(f"<h2> {text}</h2><br/>")
            elif element_type == "NarrativeText" or element_type == "UncategorizedText":
                doc_texts.append(f"<p>{text}</p>")
            elif element_type == "ListItem":
                doc_texts.append(f"<li>{text}</li>")
            elif element_type == "PageNumber":
                doc_texts.append(f"Page number: {text}")
            elif element_type == "Table":
                table_html = metadata.get("text_as_html", "")
                doc_texts.append(table_html)  # Keep the table as HTML.
            else:
                doc_texts.append(text)

        return " ".join(doc_texts)

    @mcp.tool()
    async def process_document(ctx: Context, filepath: str) -> str:
        """
        Sends the document to Unstructured for processing. 
        Returns the processed contents of the document
        
        Args:
            filepath: The local path to the document.
        """

        filepath = os.path.abspath(filepath)

        if not os.path.isfile(filepath):
            return "File does not exist"

        # Check whether Unstructured supports the file's extension.
        _, ext = os.path.splitext(filepath)
        supported_extensions = {".abw", ".bmp", ".csv", ".cwk", ".dbf", ".dif", ".doc", ".docm", ".docx", ".dot",
                                ".dotm", ".eml", ".epub", ".et", ".eth", ".fods", ".gif", ".heic", ".htm", ".html",
                                ".hwp", ".jpeg", ".jpg", ".md", ".mcw", ".mw", ".odt", ".org", ".p7s", ".pages",
                                ".pbd", ".pdf", ".png", ".pot", ".potm", ".ppt", ".pptm", ".pptx", ".prn", ".rst",
                                ".rtf", ".sdp", ".sgl", ".svg", ".sxg", ".tiff", ".txt", ".tsv", ".uof", ".uos1",
                                ".uos2", ".web", ".webp", ".wk2", ".xls", ".xlsb", ".xlsm", ".xlsx", ".xlw", ".xml",
                                ".zabw"}

        if ext.lower() not in supported_extensions:
            return "File extension not supported by Unstructured"

        client = ctx.request_context.lifespan_context.client
        file_basename = os.path.basename(filepath)

        req = operations.PartitionRequest(
            partition_parameters=shared.PartitionParameters(
                files=shared.Files(
                    content=open(filepath, "rb"),
                    file_name=filepath,
                ),
                strategy=shared.Strategy.AUTO,
            ),
        )

        try:
            res = client.general.partition(request=req)
            element_dicts = [element for element in res.elements]
            json_elements = json.dumps(element_dicts, indent=2)
            output_json_file_path = os.path.join(PROCESSED_FILES_FOLDER, f"{file_basename}.json")
            with open(output_json_file_path, "w") as file:
                file.write(json_elements)

            return json_to_text(output_json_file_path)
        except Exception as e:
            return f"The following exception happened during file processing: {e}"

    if __name__ == "__main__":
        load_environment_variables()
        # Initialize and run the server.
        mcp.run(transport='stdio')
    ```
  </Step>

  <Step>
    In the root of the project directory, create a new file named `.env`.
    Then add the following code to this new `.env` file, replacing `<your-unstructured-api-key>` with your Unstructured API key:

    ```text theme={null}
    UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
    ```
  </Step>
</Steps>

## Step 2: Configure Claude for Desktop to use the MCP server

<Steps>
  <Step>
    Open the Claude for Desktop App configuration file from the following location, for example by using Visual Studio Code:

    ```text theme={null}
    # For macOS or Linux:
    code ~/Library/Application\ Support/Claude/claude_desktop_config.json

    # For Windows:
    code $env:AppData\Claude\claude_desktop_config.json
    ```
  </Step>

  <Step>
    Add the following code to the `claude_desktop_config.json` file, and then save the file.

    In this file:

    * `"unstructured-partition-mcp"` must match the value that you set in `mcp = FastMCP("unstructured-partition-mcp" ...` in the `doc_processor.py` file.
    * "Absolute path to" `uv` must be the absolute path to the `uv` executable on your local machine.
    * "Absolute path to project directory" must be the absolute path to the project directory.
    * `mcp-unstructured-partition-demo` must match the name of the project directory that you created.
    * `"doc_processor.py"` must match the name of the Python file that you created for the MCP server.

    For macOS or Linux:

    ```json theme={null}
    {
        "mcpServers": {
            "unstructured-partition-mcp": {
                "command": "/ABSOLUTE/PATH/TO/uv",
                "args": [
                    "--directory",
                    "/ABSOLUTE/PATH/TO/PROJECT/DIRECTORY/mcp-unstructured-partition-demo/",
                    "run",
                    "doc_processor.py"
                ]
            }
        }
    }
    ```

    For Windows:

    ```json theme={null}
    {
        "mcpServers": {
            "unstructured-partition-mcp": {
                "command": "C:\\ABSOLUTE\\PATH\\TO\\uv",
                "args": [
                    "--directory",
                    "C:\\ABSOLUTE\\PATH\\TO\\PROJECT\\DIRECTORY\\mcp-unstructured-partition-demo\\",
                    "run",
                    "doc_processor.py"
                ]
            }
        }
    }
    ```
  </Step>
</Steps>

## Step 3: Run the MCP server and call it from Claude for Desktop

<Steps>
  <Step>
    Start Claude for Desktop. If it is already running, restart it.
  </Step>

  <Step>
    Ask the following question in the chat window, replacing `<absolute-path-to-local-file>` with the absolute path to the local file that you
    want Unstructured to process.

    <Note>
      This walkthrough has been tested with small files (less than 400 KB), due to Claude for Desktop's hard timeout limit of 60 seconds for MCP server responses.

      Larger files might take longer for Unstructured to process, and Claude for Desktop might time out before such files are fully processed.
    </Note>

    ```text theme={null}
    Process the document at <absolute-path-to-local-file>
    ```

    Claude for Desktop responds with information about the processed document.

    You can find the processed file's content in the path that you specified for `PROCESSED_FILES_FOLDER` in the MCP server's `doc_processor.py` file.
  </Step>
</Steps>


# Chatbot with Neo4j
Source: https://docs.unstructured.io/examplecode/tools/neo4j-chatbot





# OneDrive event triggers
Source: https://docs.unstructured.io/examplecode/tools/onedrive-events



You can use OneDrive events, such as adding new files to—or updating existing files within—OneDrive folders, to automatically run Unstructured ETL+ workflows
that rely on those folders as sources. This enables a no-touch approach to having Unstructured automatically process new and updated files in OneDrive folders as they are added or updated.

This example shows how to automate this process by adding two custom [Power Automate](https://www.microsoft.com/power-platform/products/power-automate) flows to your Microsoft 365 or Office 365 account. One flow runs
whenever a new file is detected in the specified OneDrive folder. The other flow runs whenever an existing file is updated within that folder. These flows then call the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow within your Unstructured account.

<Note>
  This example uses custom Power Automate flows that you create and maintain.
  Any issues with file detection, timing, or flow invocation could be related to your custom flows,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  flows' run histories first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* A Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online, OneDrive, and Power Automate.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* A Power Automate Premium license. [Learn how](https://learn.microsoft.com/power-platform/admin/power-automate-licensing/buy-licenses).

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

* A OneDrive source connector in your Unstructured account. [Learn how](/pipelines/sources/onedrive).

* Some available [destination connector](/pipelines/destinations/overview) in your Unstructured account.

* A workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

## Step 1: Create an app registration

In this step, you create an app registration in Microsoft Entra ID to access the Azure key vault that you will create in the next step.

1. Sign in to your [Azure portal](https://portal.azure.com).
2. In the **Search resources, services, and docs** box, enter **App registrations**, and then in the list of **Services**, click **App registrations**.
3. Click **+ New registration**.
4. Enter some **Name** for the app registration.
5. For **Supported account types**, select **Accounts in this organizational directory only**.
6. For **Redirect URI**, select **Web**, and then enter `https://localhost`.
7. Click **Register**.
8. In the sidebar, click **Overview**.
9. Expand **Essentials**, and note the values of the **Application (client) ID** and **Directory (tenant) ID** fields.
10. Next to **Client credentials**, click the **Add a certificate or secret** link.
11. On the **Client secrets** tab, click **+ New client secret**.
12. (Optional) Enter some **Description** for the client secret.
13. (Optional) Enter some non-default **Expires** value.
14. Click **Add**.
15. Note the value of the **Value** field. This is the app registration's client secret value. (Do not use the **Secret ID** value.)

## Step 2: Create an Azure key vault

In this step, you store your Unstructured API key in an Azure key vault for your Power Automate flow to access later.

1. With your Azure portal still open from the previous step, in the **Search resources, services, and docs** box, enter **Key vaults**, and then in the list of **Services**, click **Key vaults**.

2. Click **Create**.

3. On the **Basics** tab, set the following key vault settings:

   | Setting            | Suggested value           | Description                                                                                                           |
   | ------------------ | ------------------------- | --------------------------------------------------------------------------------------------------------------------- |
   | **Subscription**   | Your subscription         | The Azure subscription within which to create your new key vault.                                                     |
   | **Resource Group** | **Create new**            | After you click **Create new**, enter some name for the new resource group within which to create your new key vault. |
   | **Key vault name** | Some globally unique name | Some name that identifies your new key vault. Valid characters are `A`-`Z`, `a`-`z`, `0`-`9`, and `-`.                |
   | **Region**         | Your preferred region     | Select a region that's near you or near other services that your key vault can access.                                |
   | **Pricing tier**   | **Standard**              | Choose the pricing tier for your key vault.                                                                           |

4. Click **Review + create**.

5. Click **Create**.

6. After the deployment is complete, click **Go to resource**.

7. In the sidebar, click **Access control (IAM)**.

8. Click **+ Add > Add role assignment**.

9. On the **Role** tab, in the **Job function roles** list, select **Key Vault Administrator**.

10. Click **Next**.

11. On the **Members** tab, with **Assign access to** set to **User, group, or service principal**, click **+ Select members**.

12. Search for and select the name of the app registration that you created earlier in Step 1.

13. Search also for and select your user account.

14. Click **Select**.

15. Click **Review + assign** twice.

16. In the sidebar, expand **Objects**, and then click **Secrets**.

17. Click **+ Generate/Import**.

18. Enter some **Name** for the secret.

19. For **Secret value**, enter your Unstructured API key value.

20. Click **Create**.

## Step 3: Create the first Power Automate flow

1. Sign in to [Power Automate](https://make.powerautomate.com).

2. On the sidebar, click **Create**.

3. In the **Start from blank** area, click **Automated cloud flow**.

4. Enter some **Flow name** for your new flow.

5. For **Search all triggers**, enter `When a file is created`, and then click
   **When a file is created (OneDrive for Business)**.

6. Click **Create**.

7. In the flow designer, click the **When a file is created** node.

8. When prompted, sign in to OneDrive.

9. On the **Parameters** tab, set the following node properties:

   * For **Folder**, click the folder icon, and the select the folder that matches the value of the **Path** value for the OneDrive source connector for your workflow.
   * For **Include subfolders**, select **Yes** only if you also checked the **Recursive** checkbox for the OneDrive source connector for your workflow.

10. Below the **When a file is created** node, click the plus (**Insert a new action...**) button.

11. In the **Search for an action or connector** box, enter `Get secret`, and then under **Azure Key Vault**, select **Get secret**.

12. Set the following node properties:

    * Enter some **Connection name** for the connection.
    * For **Authentication type**, select **Service principal authentication**.
    * For **Client ID**, enter the **Application (client) ID** value of the app registration that you created earlier in Step 1.
    * For **Client secret**, enter the client secret value of the app registration that you created earlier in Step 1.
    * For **Tenant ID**, enter the **Directory (tenant) ID** value of the app registration that you created earlier in Step 1.
    * For **Key vault name**, enter the name of the key vault that you created earlier in Step 2.

13. Click **Create new**.

14. In the **Name of the secret** list, select the name of the secret that you created earlier in Step 2.

15. Below the **Get secret** node, click the plus (**Insert a new action...**) button.

16. In the **Search for an action or connector** box, enter `HTTP`, and then under **HTTP**, select **HTTP**.

17. On the **Parameters** tab, set the following node properties:

    * For **URI**, enter `<unstructured-api-url>/workflows/<workflow-id>/run`, and replace the following placeholders:

      * Replace `<unstructured-api-url>` with your Unstructured API's workflow operations value.
      * Replace `<workflow-id>` with the ID of your Unstructured workflow.

      The **URI** should now look similar to the following:

      ```text theme={null}
      https://platform.unstructuredapp.io/api/v1/workflows/11111111-1111-1111-1111-111111111111/run
      ```

    * For **Method**, select **POST**.

    * For **Headers**, enter the following:

      * `accept`, set to `application/json`
      * `unstructured-api-key`, set to your key vault value. To do this, type `/` and select **Insert dynamic content**.
        Under **Get secret**, select **body/value**.

18. In the top navigation bar, click the disk (**Save**) button.

19. Click **Back** to go to the flow's home page.

20. In the top navigation bar, if the power button shows **Turn on**, click it to change it to **Turn off** (which means that the flow is on and active).

## Step 4: Create the second Power Automate flow

1. With the flow's home page still open, on the sidebar, click **My flows**.
2. In the list of flows, rest your mouse pointer on the flow that you created earlier in Step 3, and then click the ellipsis (three dots, **More commands**) icon.
3. Click **Save as**.
4. Give the new flow a **Flow name** that is different from the original flow's name.
5. Click **Save**.
6. In the list of flows, rest your mouse pointer on the flow that you just created, and then click the pencil (**Edit**) icon.
7. In the flow designer, right click the **When a file is created** node, and then click **Delete**.
8. Click the **Add a trigger** node.
9. In the **Search for an action or connector** box, enter `When a file is modified`, and then click
   **When a file is modified (OneDrive for Business)**.
10. On the **Parameters** tab, set the following node properties:

* For **Folder**, click the folder icon, and the select the folder that matches the value of the **Path** value for the OneDrive source connector for your workflow.
* For **Include subfolders**, select **Yes** only if you also checked the **Recursive** checkbox for the OneDrive source connector for your workflow.

11. In the top navigation bar, click the disk (**Save**) button.
12. Click **Back** to go to the flow's home page.
13. In the top navigation bar, if the power button shows **Turn on**, click it to change it to **Turn off** (which means that the flow is on and active).

## Step 5: Trigger the flows and view results

1. On a separate tab in your web browser, go to the OneDrive folder that corresponds to both your Unstructured workflow's
   OneDrive source connector and the Power Automate flows' **When a file is created** and **When a file is modified** nodes.
2. Add a file to the OneDrive folder.
3. On a separate tab in your web browser, in Unstructured Pipelines for your account, click **Jobs** on the sidebar.
4. In the list of jobs, click the newly running job for your workflow. It might take a minute or so for the new job to appear in the list.
5. After the job status shows **Finished**, go to your destination location to see the results.
6. Repeat this step, but this time, update an existing file in the OneDrive folder.

## Step 6 (Optional): Turn off the flows

To stop the Power Automate flows from running automatically, go back to the web browser tab from earlier in Steps 3 and 4. Then, do the following:

1. On the sidebar, click **My flows**.
2. In the list of flows, click the name of the flow that you created earlier in Step 3.
3. With the flow's home page still open, in the top navigation bar, click the power (**Turn off**) button
   to change it to **Turn on** (which means that the flow is off and inactive).
4. Repeat this step, but this time, click the name of the flow that you created earlier in Step 4.


# PII detection
Source: https://docs.unstructured.io/examplecode/tools/pii



Personally identifiable information (PII) detection is important as part of an organization's overall strategy for
minimizing potential harm from misuse or unauthorized access to individuals' private data. Detecting PII is a first step
toward helping avoid identity theft, maintaining privacy, and building trust with customers and users. Organizations
might also need to follow various data protection regulations, making PII detection a crucial part of the
organization's legal compliance framework.

This hands-on example walkthrough demonstrates how to use the [Microsoft Presidio SDK](https://microsoft.github.io/presidio/)
to identify and then redact PII in Unstructured JSON output files. Presidio can identify
and redact or anonymize entities in text and images such as credit card numbers, names, locations, social security numbers,
bitcoin wallets, US phone numbers, financial data, and more.

In this walkthrough, you will use Python code to connect to a folder within an Amazon S3 bucket that already contains a
collection of Unstructured JSON output files. For each file, your code will use Presidio to identify PII that matches
specific patterns and then redact the identified PII. Your code will then write the Unstrucutred JSON output files' contents
that contain the redacted PII to a separate folder within an S3 bucket. You can then compare the output JSON files generated by
Unstructured to the redacted data generated by Presidio and see the impact of using Presidio for PII detection.

*Redaction* directly removes or obscures PII, such as replacing names with placeholders, or blacking out sensitive text.
The example code from this walkthrough replaces the detected PII characters within text strings with placeholder text. Another technique, *anonymization*,
can involve redaction, but also includes techniques such as *generalization* (for example, replacing specific dates
with age ranges), *suppression* (removing entire data fields), and *data masking* (replacing data with random values
while preserving the format). Presidio supports both redaction and anonymization. Although the term "anonymize" is
visible throughout this walkthrough's example code, Presidio's available anonymization techniques are not explored here—only redaction.

<Note>
  The Microsoft Presidio SDK is not offered, maintained, or supported by Unstructured. For questions or issues
  related to Presidio, see the following resources:

  * For general discussions, use the [discussion board](https://github.com/microsoft/presidio/discussions) in the Presidio repository on GitHub.
  * For questions or issues, file an [issue](https://github.com/microsoft/presidio/issues) in the Presidio repository on GitHub.
  * For other matters, email [presidio@microsoft.com](mailto:presidio@microsoft.com).

  The example code in this walkthrough is provided as a general reference only. It is not intended to substitute
  for a complete PII detection strategy.

  The example code in this walkthrough is not guaranteed to detect and redact all possible PII. For instance, the code
  looks for PII only in text strings. It does not look for PII in non-text fields such as `image_base64` and `orig_elements`
  within Unstructured metadata, and it does not look for PII in images.
</Note>

## Requirements

To use this example, you will need:

* An Unstructured account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/pipelines/overview#how-am-i-billed%3F).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

* A set of one or more Unstructured JSON output files that have been generated by Unstructured and stored in a folder within an
  Amazon S3 bucket that you have access to. One way to generate these files is to use an Unstructured workflow that
  relies on an S3 destination connector to store these Unstructured JSON output files. Learn how to [create an S3 destination connector](/pipelines/destinations/s3) and
  [create a custom workflow](/pipelines/workflows#create-a-custom-workflow) that uses your S3 destination connector.

* Python installed on your local development machine.

## Create and run the Python code

1. In your local Python virtual environment, install the following libraries:

   * `boto3`
   * `presidio_analyzer`
   * `presidio_anonymizer`

   For example, if you are using [uv](https://docs.astral.sh/uv/), you can install these libraries into your local `uv`
   virtual environment with the following command:

   ```bash theme={null}
   uv add boto3 presidio_analyzer presidio_anonymizer
   ```

2. In your local Python virtual environment, install the appropriate natural language processing (NLP) models for
   [spaCy](https://spacy.io/), which Presidio relies on for various internal tasks related to named entity recognition (NER) and
   PII identification.

   To find the appropriate model for your use case, do the following:

   a. Go to [spaCy Trained Models & Pipelines](https://spacy.io/models).<br />
   b. On the sidebar, click your target language, for example **English**.<br />
   c. Click the model you want to use, for example **en\_core\_web\_lg**.<br />
   d. Click **Release details**.<br />
   e. At the bottom of the release details page, in the **Assets** section, right-click the filename ending in `.whl`, for
   example **en\_core\_web\_lg-3.8.0-py3-none-any.whl**, and select **Copy Link Address** from the context menu.<br />
   f. From the release details page, copy the URL from your web browser's address bar.<br />
   g. Install the model into your local Python virtual environment by using the model's name and the URL that you just copied.
   For example, if you are using `uv`, you can install the preceding model with a command such as the following:<br />

   ```bash theme={null}
   uv pip install en_core_web_lg@https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl
   ```

3. [Set up Boto3 credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for your AWS account.
   The following steps assume you have set up your Boto3 credentials from outside of the following code, such as setting
   [environment variables](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#environment-variables) or
   configuring a [shared credentials file](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#shared-credentials-file),

   One approach to getting and setting up Boto3 credentials is to [create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey)
   and then use the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
   to [set up your credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-methods) on your local development machine.

   <iframe title="YouTube video player" />

4. Add the following code to a Python script file in your virtual environment, replacing the following placeholders:

   * Replace `<input-bucket-name>` with the name of the Amazon S3 bucket that contains your original Unstructured JSON files. This is the same
     bucket that you used for your S3 destination connector.
   * Replace `<input-folder-prefix>` with the path to the folder within the input bucket that contains your original Unstructured JSON files.
   * Replace `<output-bucket-name>` with the name of the S3 bucket that will contain copies of the contents of your Unstructured JSON files,
     with the redacted content within those files' copies. This can be the same bucket as the input bucket, or a different bucket.
   * Replace `<output-folder-prefix>` with the path to the folder within the output bucket that will contain copies of the contents of your Unstructured
     JSON files, with the redacted content within those files' copies. This must not be the same folder as the input folder.
   * Replace `<bucket-region-short-id>` with the short ID of the region where your buckets are located, for example `us-east-1`.

   For the `operators` variable, a list of operators for built-in Presidio entities is specified. These operators look for common entities such as
   credit card numbers, email addresses, phone numbers, and more. You can remove any entities from this list that you do not
   want your code to look for. You can also add operators to this list for additional
   [built-in entities](https://microsoft.github.io/presidio/supported_entities/) that you want your code to also look for. And you can also
   [add your own custom entities](https://microsoft.github.io/presidio/analyzer/adding_recognizers/) to this list.

   ```python theme={null}
   import boto3
   import json

   from presidio_analyzer import AnalyzerEngine
   from presidio_anonymizer import AnonymizerEngine
   from presidio_anonymizer.entities import OperatorConfig

   operators={
       "CREDIT_CARD": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_CREDIT_CARD>'}),
       "CRYPTO": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_CRYPTO>'}),
       "EMAIL_ADDRESS": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_EMAIL_ADDRESS>'}),
       "IBAN_CODE": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_IBAN_CODE>'}),
       "IP_ADDRESS": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_IP_ADDRESS>'}),
       "NRP": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_NRP>'}),
       "LOCATION": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_LOCATION>'}), 
       "PERSON": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_PERSON>'}),
       "PHONE_NUMBER": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_PHONE_NUMBER>'}),
       "MEDICAL_LICENSE": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_MEDICAL_LICENSE>'}),
       "URL": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_URL>'}),
       "US_BANK_NUMBER": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_US_BANK_NUMBER>'}),
       "US_DRIVER_LICENSE": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_US_DRIVER_LICENSE>'}),
       "US_ITIN": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_US_ITIN>'}),
       "US_PASSPORT": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_US_PASSPORT>'}),
       "US_SSN": OperatorConfig(operator_name="replace", params={'new_value': '<REDACTED_US_SSN>'})
   }

   # Recursively check for string values in the provided JSON object (in this case, 
   # the "metadata" field of the JSON object) and redact them 
   # as appropriate.
   def check_string_values(obj, analyzer, anonymizer):
       if isinstance(obj, dict):
           for key, value in obj.items():
               # Skip analyzing Base64-encoded fields.
               if key == 'image_base64' or key == 'orig_elements':
                   pass
               elif isinstance(value, str):
                   anonymized_results = anonymizer.anonymize(
                       text=value,
                       analyzer_results=analyzer.analyze(text=value, language="en"),
                       operators=operators
                   )
                   value = anonymized_results.text
               # Recurse through nested "metadata" fields.
               elif isinstance(value, dict):
                   check_string_values(value, analyzer, anonymizer)
               # Skip analyzing non-string fields.
               else:
                   pass
       return obj

   def main():
       s3_input_bucket_name    = '<input-bucket-name>'
       s3_input_folder_prefix  = '<input-folder-prefix>'
       s3_output_bucket_name   = '<output-bucket-name>'
       s3_output_folder_prefix = '<output-folder-prefix>'
       s3_bucket_region        = '<bucket-region-short-id>'

       s3_client = boto3.client('s3')

       # Load the JSON files from the input folder.
       # Normalize the input folder prefix to ensure it ends with '/'.
       if not s3_input_folder_prefix.endswith('/'):
           s3_input_folder_prefix += '/'

       paginator = s3_client.get_paginator('list_objects_v2')
       page_iterator = paginator.paginate(
           Bucket=s3_input_bucket_name, 
           Prefix=s3_input_folder_prefix
       )
       files = []

       # Get the list of file keys from the input folder to analyze.
       # A file's key is the full path to the file within the bucket.
       # For example, if the input folder's name is "original" and the 
       # input file's name is "file1.json", the file's key is 
       # "original/file1.json".
       # There could be multiple "pages" of file listings available, 
       # so each of these "pages" must be looped through, so that 
       # no files are missed.
       for page in page_iterator:
           # "Contents" is missing if the folder is empty or the 
           # intended prefix is not found.
           if 'Contents' in page:
               for obj in page['Contents']:
                   key = obj['Key']
                   if not key.endswith('/'):  # Skip if it's a folder placeholder.
                       files.append(key)
                       print(f"Found file: {s3_input_bucket_name}/{key}")

       analyzer = AnalyzerEngine()
       anonymizer = AnonymizerEngine()
       s3_resource = boto3.resource('s3')

       # For each JSON file to analyze, load the JSON data.
       for key in files:
           print(f"Analyzing file: {s3_input_bucket_name}/{key}")
           content_object = s3_resource.Object(
               bucket_name=s3_input_bucket_name, 
               key=key 
           )

           file_content = content_object.get()['Body'].read().decode('utf-8')  # Bytes to text.
           json_data = json.loads(file_content) # Text to JSON.

           # For each element in the JSON data...
           for element in json_data:
               print(f"    Analyzing element with ID: {element['element_id']} in file {s3_input_bucket_name}/{key}")
               # If there is a "text" field...
               if 'text' in element:
                   # ...get the text content...
                   text_element = element['text']
                   # ...and analyze and redact the text content as appropriate.
                   anonymized_results = anonymizer.anonymize(
                       text=text_element,
                       analyzer_results=analyzer.analyze(text=text_element, language="en"),
                       operators=operators
                   )
                   element['text'] = anonymized_results.text
               # If there is a "metadata" field...
               if 'metadata' in element:
                   # ...get the metadata content...
                   metadata_element = element['metadata']
                   # ...and analyze and redact the metadata content as appropriate.
                   element['metadata'] = check_string_values(metadata_element, analyzer, anonymizer)

           # Get the filename from the key.
           filename = key.split(s3_input_folder_prefix)[1]

           # Normalize the output folder prefix to ensure it ends with '/'.
           if not s3_output_folder_prefix.endswith('/'):
               s3_output_folder_prefix += '/'
           
           # Then save the JSON data with its redactions to the output folder.
           print(f"Saving file: {s3_output_bucket_name}/{s3_output_folder_prefix}{filename}")
           s3_client.put_object(
               Bucket=s3_output_bucket_name,
               Key=f"{s3_output_folder_prefix}{filename}",
               Body=json.dumps(obj=json_data, indent=4).encode('utf-8')
           )

   if __name__ == "__main__":
       main()
   ```

5. Run the Python script.

6. Go to the output folder in S3 and explore the generated files, searching for the `<REDACTED_` placeholders in the generated files' contents.


# Amazon S3 event triggers
Source: https://docs.unstructured.io/examplecode/tools/s3-events



You can use Amazon S3 events, such as adding new files to—or updating existing files within—S3 buckets, to automatically run Unstructured ETL+ workflows
that rely on those buckets as sources. This enables a no-touch approach to having Unstructured automatically process new and updated files in S3 buckets as they are added or updated.

This example shows how to automate this process by adding an [AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/concepts-basics.html#gettingstarted-concepts-function) to your AWS account. This function runs
whenever a new file is added to—or an existing file is updated within—the specified S3 bucket. This function then calls the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow in your Unstructured account.

<Note>
  This example uses a custom AWS Lambda function that you create and maintain.
  Any issues with file detection, timing, or function invocation could be related to your custom function,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  function's Amazon CloudWatch logs first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

* An S3 source connector in your Unstructured account. [Learn how](/pipelines/sources/s3).

* Some available [destination connector](/pipelines/destinations/overview) in your Unstructured account.

* A workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

## Step 1: Create the Lambda function

1. Sign in to the AWS Management Console for your account.
2. Browse to and open the **Lambda** console.
3. On the sidebar, click **Functions**.
4. Click **Create function**.
5. Select **Author from scratch**.
6. For **Function name**, enter a name for your function, such as `RunUnstructuredWorkflow`.
7. For **Runtime**, select **Node.js 22.x**.
8. For **Architecture**, select **x86\_64**.
9. Under **Permissions**, expand **Change default execution role**, and make sure **Create a new role with basic Lambda permissions** is selected.
10. Click **Create function**. After the function is created, the function's code and configuration settings page appears.

## Step 2: Add code to the function

1. With the function's code and configuration settings page open from the previous step, click the **Code** tab.

2. In the **Code source** tile, replace the contents of the `index.mjs` file with the following code instead.

   If the `index.mjs` file is not visible, do the following:

   1. Show the **Explorer**: on the sidebar, click **Explorer**.
   2. In the **Explorer** pane, expand the function name.
   3. Click to open the **index.mjs** file.

   Here is the code for the `index.mjs` file:

   ```javascript theme={null}
   import https from 'https';

   export const handler = async (event) => {
     const apiUrl = process.env.UNSTRUCTURED_API_URL;
     const apiKey = process.env.UNSTRUCTURED_API_KEY;

     if (!apiUrl || !apiKey) {
       throw new Error('Missing UNSTRUCTURED_API_URL or UNSTRUCTURED_API_KEY environment variable or both.');
     }

     const url = new URL(apiUrl);

     const options = {
       hostname: url.hostname,
       path: url.pathname,
       method: 'POST',
       headers: {
         'accept': 'application/json',
         'unstructured-api-key': apiKey
       }
     };

     const postRequest = () => new Promise((resolve, reject) => {
       const req = https.request(options, (res) => {
         let responseBody = '';
         res.on('data', (chunk) => { responseBody += chunk; });
         res.on('end', () => {
           resolve({ statusCode: res.statusCode, body: responseBody });
         });
       });
       req.on('error', reject);
       req.end();
     });

     try {
       const response = await postRequest();
       console.log(`POST status: ${response.statusCode}, body: ${response.body}`);
     } catch (error) {
       console.error('Error posting to endpoint:', error);
     }

     return {
       statusCode: 200,
       body: JSON.stringify('Lambda executed successfully')
     };
   };
   ```

3. In **Explorer**, expand **Deploy (Undeployed Changes)**.

4. Click **Deploy**.

5. Click the **Configuration** tab.

6. On the sidebar, click **Environment variables**.

7. Click **Edit**.

8. Click **Add environment variable**.

9. For **Key**, enter `UNSTRUCTURED_API_URL`.

10. For **Value**, enter `<unstructured-api-url>/workflows/<workflow-id>/run`. Replace the following placeholders:

    * Replace `<unstructured-api-url>` with your Unstructured API's workflow operations value.
    * Replace `<workflow-id>` with the ID of your Unstructured workflow.

    The **Value** should now look similar to the following:

    ```text theme={null}
    https://platform.unstructuredapp.io/api/v1/workflows/11111111-1111-1111-1111-111111111111/run
    ```

11. Click **Add environment variable** again.

12. For **Key**, enter `UNSTRUCTURED_API_KEY`.

13. For **Value**, enter your Unstructure API key value.

14. Click **Save**.

## Step 3: Create the function trigger

1. Browse to and open the S3 console.

2. Browse to and open the S3 bucket that corresponds to your S3 source connector. The bucket's settings page appears.

3. Click the **Properties** tab.

4. In the **Event notifications** tile, click **Create event notification**.

5. In the **General configuration** tile, enter a name for your event notification, such as `UnstructuredWorkflowNotification`.

6. (Optional) For **Prefix**, enter any prefix to limit the Lambda function's scope to only the specified prefix. For example, to limit the scope to only
   the `input/` folder within the S3 bucket, enter `input/`.

   <Warning>
     AWS does not recommend reading from and writing to the same S3 bucket because of the possibility of accidentally running Lambda functions in loops.
     However, if you must read from and write to the same S3 bucket, AWS strongly recommends specifying a **Prefix** value. [Learn more](https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html).
   </Warning>

7. (Optional) For **Suffix**, enter any file extensions to limit the Lambda function's scope to only the specified file extensions. For example, to limit the scope to only
   files with the `.pdf` extension, enter `.pdf`.

8. In the **Event types** tile, check the box titled **All object create events (s3:ObjectCreated:\*)**.

9. In the **Destination** tile, select **Lambda function** and **Choose from your Lambda functions**.

10. In the **Lambda function** tile, select the Lambda function that you created earlier in Step 1.

11. Click **Save changes**.

## Step 4: Trigger the function

1. With the S3 bucket's settings page open from the previous step, click the **Objects** tab.
2. If you specified a **Prefix** value earlier in Step 3, then click to open the folder that corresponds to your **Prefix** value.
3. Click **Upload**, and then follow the on-screen instructions to upload a file to the bucket's root. If, however, you clicked to open the folder that corresponds to your **Prefix** value instead, then follow the on-screen instructions to upload a file to that folder instead.

## Step 5: View the trigger results

1. In Unstructured Pipelines for your account, click **Jobs** on the sidebar.
2. In the list of jobs, click the newly running job for your workflow.
3. After the job status shows **Finished**, go to your destination location to see the results.

## Step 6 (Optional): Delete the trigger

1. To stop the function from automatically being triggered whenever you add new files to—or update existing files within—the S3 bucket, browse to and open the S3 console.
2. Browse to and open the bucket that corresponds to your S3 source connector. The bucket's settings page appears.
3. Click the **Properties** tab.
4. In the **Event notifications** tile, check the box next to the name of the event notification that you added earlier in Step 3.
5. Click **Delete**, and then click **Confirm**.


# Amazon S3 Vectors
Source: https://docs.unstructured.io/examplecode/tools/s3-vectors



[Amazon S3 Vectors](https://aws.amazon.com/s3/features/vectors/) is a durable vector storage solution that can greatly reduce the total cost of uploading, storing,
and querying vectors. S3 Vectors is a cloud object store with native support to store large vector datasets and
provide subsecond query performance. This makes it more affordable for businesses to store AI-ready data at massive scale.

This hands-on example walkthrough demonstrates how to use Amazon S3 Vectors with Unstructured. In this walkthrough, you will:

1. Create an S3 vector bucket.
2. Create a vector index in the bucket.
3. Add the contents of one or more source JSON output files that have been generated by Unstructured to the vector index.
4. Query the vector index against the contents of the source JSON output files that were added.

## Requirements

To use this example, you will need:

* A set of one or more JSON output files that have been generated by Unstructured and stored somewhere on your local development machine. For maximum compatibility with this example, these files must contain vector embeddings that were
  generated by Amazon Bedrock, by using the **Titan Text Embeddings V2** (`amazon.titan-embed-text-v2:0`) embedding model, with 1024 dimensions. To get these files, you will need:

  * An Unstructured account, as follows:

    1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
       After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

       <Note>
         To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/pipelines/overview#how-am-i-billed%3F).
       </Note>

    2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

       <Note>
         For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
         or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
       </Note>

  * A workflow that generates vector embeddings and adds them to the JSON output files. Learn how to [create a custom workflow](/pipelines/workflows#create-a-custom-workflow) and [add an Embedder node](/concepts/embedding#generate-embeddings) to that workflow.

    <Warning>
      The [destination connector](/pipelines/destinations/overview) for your worklow must generate JSON output files. These include destination connectors for file storage connectors such as
      Databricks Volumes, Google Cloud Storage, OneDrive, and S3. Destination connectors for databases such as Elasticsearch, Kafka, and MongoDB, and vector stores such as Astra DB, Pinecone, and Weaviate, do not generate JSON output files.
    </Warning>

  * After your workflow generates the JSON output files, you must copy them from your workflow's destination location over to some location on your local development machine for access.

* Python installed on your local development machine.

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

## Step 1: Create the S3 vector bucket

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).
2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).
3. On the sidebar, click **Vector bucket**.
4. Click **Create vector bucket**.
5. For **Vector bucket name**, enter some name for your bucket.
6. For **Encryption**, select an encryption method, or leave the default.
7. Click **Create vector bucket**.

## Step 2: Add the vector index to the bucket

1. With the list of vector buckets showing from the previous step, click the name of the bucket that you just created.
2. Click **Create vector index**.
3. For **Vector index name**, enter some name for your index.
4. For **Dimension** enter the number of dimensions that Unstructured generated for your vector embeddings. For example,
   for the **Titan Text Embeddings V2** (`amazon.titan-embed-text-v2:0`) embedding model, enter `1024`. If you are not sure how many dimensions to enter,
   see your workflow's **Embedder** node settings.
5. Select the appropriate **Distance metric** for your embedding model. For example, for the
   **Titan Text Embeddings V2** (`amazon.titan-embed-text-v2:0`) embedding model, select **Cosine**. If you are not sure which distance metric to use,
   see your embedding model's documentation.
6. Expand **Additional settings**.
7. Within **Metadata configuration**, under **Non-filterable metadata**, click **Add key**.
8. For **Key**, enter `text`. This allows you to query the vector index by the `text` field within each object that will be coming over into the index from the JSON output files.
9. Click **Create vector index**.
10. After the vector index is created, copy the value of the index's **Amazon Resource Name (ARN)**, as you will need it in
    later steps. This ARN takes the format `arn:aws:s3vectors:<region-id>:<account>:bucket/<bucket-name>/index/<index-name>`.

## Step 3: Add the source JSON output files' contents to the vector index

1. In your local Python virtual environment, install the `boto3` and `uuid` libraries.

2. [Set up Boto3 credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) for your AWS account.
   The following steps assume you have set up your Boto3 credentials from outside of the following code, such as setting
   [environment variables](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#environment-variables) or
   configuring a [shared credentials file](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#shared-credentials-file),

   One approach to getting and setting up Boto3 credentials is to [create an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey)
   and then use the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
   to [set up your credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-methods) on your local development machine.

   <iframe title="YouTube video player" />

3. Add the following code to a Python script file in your virtual environment, replacing the following placeholders:

   * Replace `<source-json-file-path>` with the path to the directory that contains your JSON output files.
   * Replace `<index-arn>` with the ARN of the vector index that you created previously in Step 2.
   * Replace `<index-region-short-id>` with the short ID of the region where your vector index is located, for example `us-east-1`.

   ```python theme={null}
   import boto3
   import os
   import json
   import uuid

   source_json_file_path = '<source-json-file-path>'
   index_arn             = '<index-arn>'
   index_region_short_id = '<index-region-short-id>'
   s3vectors             = boto3.client('s3vectors', region_name=index_region_short_id)
   num_vectors           = 0
   verbose               = True # Set to False to only print final results.

   # For each JSON file in the source directory...
   for filename in os.listdir(source_json_file_path):
       if filename.endswith('.json'):
           # ...read the JSON file, and then...
           with open(source_json_file_path + filename, 'r') as f:
               file = json.load(f)
               # ...for each object in the Unstructured-formatted JSON array...
               for object in file:
                   # ...add the object's text and vectors to the S3 vector index.
                   # Use the following format:

                   # {
                   #     "key": "<random-uuid>",
                   #     "data": {
                   #       "float32": <embedding-array>
                   #     }, 
                   #     "metadata": {
                   #         "text": "<text-field>"
                   #     }
                   # }
                   json_object = {}
                   json_object['key'] = str(uuid.uuid4())
                   json_object['data'] = {}
                   json_object['metadata'] = {}

                   # If the object has no text, do not add it to the 
                   # vector index, and move on to the next object.
                   if 'text' in object:
                       json_object['metadata']['text'] = object['text']
                   else:
                       if verbose:
                           print(f"Skipping object with source element ID {object['id']} as it has no text")
                       continue

                   # If the object has no embeddings, do not add it to the 
                   # vector index either, and move on to the next object.
                   if 'embeddings' in object:
                       json_object['data']['float32'] = object['embeddings']
                   else:
                       if verbose:
                           print(f"Skipping object with source element ID {object['element_id']} as it has no embeddings")
                       continue

                   # Add the object to the vector index.
                   s3vectors.put_vectors(
                       indexArn=index_arn,
                       vectors=[json_object]
                   )

                   if verbose:
                       print(f"Added a vector entry and assigned it the internal ID {json_object['key']}. " + 
                             f"First 20 characters of the text: {object['text'][:20]}")
                   
                   num_vectors += 1

   print(f"Added {num_vectors} vector entries to the vector index.")
   ```

4. Run the script to add the JSON output files' contents to the vector index. Each object in each JSON output file
   is added as a vector entry in the vector index.

## Step 4: Query the vector index

1. In your local Python virtual environment, install the `numpy` library.

2. Add the following code to another Python script file in your virtual environment, replacing the following placeholders:

   * Replace `<index-arn>` with the ARN of the vector index that you created previously in Step 2.
   * Replace `<index-region-short-id>` with the short ID of the region where your vector index is located, for example `us-east-1`.
   * Replace `<sentence-to-embed>` with the search text that you want to embed for the query.

   ```python theme={null}
   import boto3
   import json
   import numpy as np

   index_arn             = '<index-arn>'
   index_region_short_id = '<index-region-short-id>'
   client                = boto3.client('s3vectors', region_name=index_region_short_id)

   # The sentence to embed.
   sentence = '<sentence-to-embed>'

   # Generate embeddings for the sentence to embed.
   model_id    = 'amazon.titan-embed-text-v2:0'
   bedrock     = boto3.client('bedrock-runtime', region_name=index_region_short_id)
   body        = {'inputText': f"{sentence}"}
   json_string = json.dumps(body)
   json_bytes  = json_string.encode('utf-8')

   response = bedrock.invoke_model(
       modelId=model_id,
       body=json_bytes,
       contentType='application/json',
       accept='application/json'
   )

   # Get the embeddings for the sentence and prepare them for the query.
   response_body = json.loads(response['body'].read().decode())
   embedding     = response_body['embedding']
   embedding     = np.array(embedding, dtype=np.float32).tolist()

   # Run the query.
   response = client.query_vectors(
       indexArn=index_arn,
       topK=5,
       queryVector={'float32': embedding}
   )

   print(f"Original search query: {sentence}")
   print(f"\nTop 5 results by similarity search...")
   print('-----')

   for vector in response['vectors']:
       response = client.get_vectors(
           indexArn=index_arn,
           keys=[vector['key']],
           returnData=True,
           returnMetadata=True
       )

       for vector in response['vectors']:
           print(vector['metadata']['text'])
           print('-----')
   ```

3. Run the script to query the vector index and see the query results.

## Appendix: Additional operations

Use the following code examples to perform additional vector index and vector bucket operations.

### List all entries in a vector index

Replace the following placeholders:

* Replace `<index-arn>` with the ARN of the vector index that you created earlier in Step 2.
* Replace `<index-region-short-id>` with the short ID of the region where your vector index is located, for example `us-east-1`.

```python theme={null}
import boto3

index_arn             = '<index-arn>'
index_region_short_id = '<index-region-short-id>'
client                = boto3.client('s3vectors', region_name=index_region_short_id)
num_vectors           = 0
next_token            = None
verbose               = True # Set to False to only print final results.

# List all of the vectors in the S3 vector index.
# Vectors are fetched by "page", so loop through the index in pages.
while True:
    kwargs = {
        'indexArn': index_arn,
        'returnData': True,
        'returnMetadata': True
    }

    if next_token:
        kwargs['nextToken'] = next_token

    response = client.list_vectors(**kwargs)

    for vector in response['vectors']:
        if verbose:
            print(f"Found vector entry with internal ID {vector['key']}. " +
                  f"First 20 characters of the text: {vector['metadata']['text'][:20]}")
        num_vectors += 1
    
    if 'nextToken' in response:
        next_token = response['nextToken']
    else:
        break

print(f"Total number of vector entries found: {num_vectors}")
```

### Delete all entries from a vector index

<Warning>
  This operation will permanently delete all vector entries in the vector index. This operation cannot be undone.
</Warning>

Replace the following placeholders:

* Replace `<index-arn>` with the ARN of the vector index that you created earlier in Step 2.
* Replace `<index-region-short-id>` with the short ID of the region where your vector index is located, for example `us-east-1`.

```python theme={null}
import boto3

index_arn             = '<index-arn>'
index_region_short_id = '<index-region-short-id>'
client                = boto3.client('s3vectors', region_name=index_region_short_id)
num_vectors           = 0
next_token            = None
verbose               = True # Set to False to only print final results.

# Delete all of the vectors in the S3 vector index.
# Vectors are deleted by "page", so loop through the index in pages.
while True:
    kwargs = {
        'indexArn': index_arn,
        'returnData': True,
        'returnMetadata': True
    }

    if next_token:
        kwargs['nextToken'] = next_token

    response = client.list_vectors(**kwargs)

    for vector in response['vectors']:
        # Delete each vector by its key.
        for vector in response['vectors']:
            if verbose:
                print(f"Deleting vector entry with internal ID {vector['key']}. " + 
                    f"First 20 characters of the text: {vector['metadata']['text'][:20]}")
                
            client.delete_vectors(
                indexArn=index_arn,
                keys=[vector['key']]
            )

            num_vectors += 1

    if 'nextToken' in response:
        next_token = response['nextToken']
    else:
        break

print(f"Deleted {num_vectors} vector entries from the vector index.")
```

### Delete a vector index

<Warning>
  This operation will permanently delete a vector index. This operation cannot be undone.
</Warning>

Replace the following placeholders:

* Replace `<index-arn>` with the ARN of the vector index that you created earlier in Step 2.
* Replace `<index-region-short-id>` with the short ID of the region where your vector index is located, for example `us-east-1`.

```python theme={null}
import boto3

index_arn             = '<index-arn>'
index_region_short_id = '<index-region-short-id>'
client                = boto3.client('s3vectors', region_name=index_region_short_id)

client.delete_index(
    indexArn=index_arn
)
```

### Delete a vector bucket

<Warning>
  This operation will permanently delete a vector bucket. This operation cannot be undone.
</Warning>

Replace the following placeholders:

* Replace `<bucket-arn>` with the ARN of the vector bucket that you created earlier in Step 1. To get the ARN, do the following:

  1. In the Amazon S3 console, on the sidebar, click **Vector buckets**.
  2. Next to the name of the vector bucket that you want to delete, click the copy button next to the bucket's **Amazon Resource Name (ARN)**.

* Replace `<index-region-short-id>` with the short ID of the region where your vector bucket is located, for example `us-east-1`.

```python theme={null}
import boto3

bucket_arn            = '<bucket-arn>'
index_region_short_id = '<index-region-short-id>'
client                = boto3.client('s3vectors', region_name=index_region_short_id)

client.delete_vector_bucket(
    vectorBucketArn=bucket_arn
)
```


# SharePoint event triggers
Source: https://docs.unstructured.io/examplecode/tools/sharepoint-events



You can use SharePoint events, such as adding new files to—or updating existing files within—SharePoint document libraries, to automatically run Unstructured ETL+ workflows
that rely on those document libraries as sources. This enables a no-touch approach to having Unstructured automatically process new and updated files in SharePoint document libraries as they are added or updated.

This example shows how to automate this process by adding a custom [Power Automate](https://www.microsoft.com/power-platform/products/power-automate) flow to your Microsoft 365 or Office 365 account. This flow runs
whenever a new or updated file is detected in the specified SharePoint document library. This flow then calls the [Unstructured API's workflow operations](/api-reference/workflow/overview) to automatically run the
specified corresponding Unstructured ETL+ workflow within your Unstructured account.

<Note>
  This example uses a custom Power Automate flow that you create and maintain.
  Any issues with file detection, timing, or flow invocation could be related to your custom flow,
  rather than with Unstructured. If you are getting unexpected or no results, be sure to check your custom
  flow's run history first for any informational and error messages.
</Note>

## Requirements

To use this example, you will need the following:

* A Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online, OneDrive, and Power Automate.
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* A Power Automate Premium license. [Learn how](https://learn.microsoft.com/power-platform/admin/power-automate-licensing/buy-licenses).

* An Unstructured account, and an Unstructured API key for your account, as follows:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

* The Unstructured API's workflow operations URL for your account, as follows:

  1. In Unstructured Pipelines, click **API Keys** on the sidebar.<br />
  2. Note the value of the **Unstructured API's workflow operations** field.

* A SharePoint source connector in your Unstructured account. [Learn how](/pipelines/sources/sharepoint).

* Some available [destination connector](/pipelines/destinations/overview) in your Unstructured account.

* A workflow that uses the preceding source and destination connectors. [Learn how](/pipelines/workflows).

## Step 1: Create an app registration

In this step, you create an app registration in Microsoft Entra ID to access the Azure key vault that you will create in the next step.

1. Sign in to your [Azure portal](https://portal.azure.com).
2. In the **Search resources, services, and docs** box, enter **App registrations**, and then in the list of **Services**, click **App registrations**.
3. Click **+ New registration**.
4. Enter some **Name** for the app registration.
5. For **Supported account types**, select **Accounts in this organizational directory only**.
6. For **Redirect URI**, select **Web**, and then enter `https://localhost`.
7. Click **Register**.
8. In the sidebar, click **Overview**.
9. Expand **Essentials**, and note the values of the **Application (client) ID** and **Directory (tenant) ID** fields.
10. Next to **Client credentials**, click the **Add a certificate or secret** link.
11. On the **Client secrets** tab, click **+ New client secret**.
12. (Optional) Enter some **Description** for the client secret.
13. (Optional) Enter some non-default **Expires** value.
14. Click **Add**.
15. Note the value of the **Value** field. This is the app registration's client secret value. (Do not use the **Secret ID** value.)

## Step 2: Create an Azure key vault

In this step, you store your Unstructured API key in an Azure key vault for your Power Automate flow to access later.

1. With your Azure portal still open from the previous step, in the **Search resources, services, and docs** box, enter **Key vaults**, and then in the list of **Services**, click **Key vaults**.

2. Click **Create**.

3. On the **Basics** tab, set the following key vault settings:

   | Setting            | Suggested value           | Description                                                                                                           |
   | ------------------ | ------------------------- | --------------------------------------------------------------------------------------------------------------------- |
   | **Subscription**   | Your subscription         | The Azure subscription within which to create your new key vault.                                                     |
   | **Resource Group** | **Create new**            | After you click **Create new**, enter some name for the new resource group within which to create your new key vault. |
   | **Key vault name** | Some globally unique name | Some name that identifies your new key vault. Valid characters are `A`-`Z`, `a`-`z`, `0`-`9`, and `-`.                |
   | **Region**         | Your preferred region     | Select a region that's near you or near other services that your key vault can access.                                |
   | **Pricing tier**   | **Standard**              | Choose the pricing tier for your key vault.                                                                           |

4. Click **Review + create**.

5. Click **Create**.

6. After the deployment is complete, click **Go to resource**.

7. In the sidebar, click **Access control (IAM)**.

8. Click **+ Add > Add role assignment**.

9. On the **Role** tab, in the **Job function roles** list, select **Key Vault Administrator**.

10. Click **Next**.

11. On the **Members** tab, with **Assign access to** set to **User, group, or service principal**, click **+ Select members**.

12. Search for and select the name of the app registration that you created earlier in Step 1.

13. Search also for and select your user account.

14. Click **Select**.

15. Click **Review + assign** twice.

16. In the sidebar, expand **Objects**, and then click **Secrets**.

17. Click **+ Generate/Import**.

18. Enter some **Name** for the secret.

19. For **Secret value**, enter your Unstructured API key value.

20. Click **Create**.

## Step 3: Create a Power Automate flow

1. Sign in to [Power Automate](https://make.powerautomate.com).

2. On the sidebar, click **Create**.

3. In the **Start from blank** area, click **Automated cloud flow**.

4. Enter some **Flow name** for your new flow.

5. For **Search all triggers**, enter `When a file is created or modified (properties only)`, and then click
   **When a file is created or modified (properties only) (SharePoint)**.

6. Click **Create**.

7. In the flow designer, click the **When a file is created or modified (properties only)** node.

8. On the **Parameters** tab, set the following node properties:

   * For **Site Address**, enter the **Site URL** value of the SharePoint source connector for your workflow.
   * For **Library Name**, enter the **Documents Library** value of the SharePoint source connector for your workflow.
   * (Optional) If you specified a **Path** value for the SharePoint source connector for your workflow, select **Folder** in the **Advanced parameters** list and then select the corresponding folder name.

9. Below the **When a file is created or modified (properties only)** node, click the plus (**Insert a new action...**) button.

10. In the **Search for an action or connector** box, enter `Get secret`, and then under **Azure Key Vault**, select **Get secret**.

11. Set the following node properties:

    * Enter some **Connection name** for the connection.
    * For **Authentication type**, select **Service principal authentication**.
    * For **Client ID**, enter the **Application (client) ID** value of the app registration that you created earlier in Step 1.
    * For **Client secret**, enter the client secret value of the app registration that you created earlier in Step 1.
    * For **Tenant ID**, enter the **Directory (tenant) ID** value of the app registration that you created earlier in Step 1.
    * For **Key vault name**, enter the name of the key vault that you created earlier in Step 2.

12. Click **Create new**.

13. In the **Name of the secret** list, select the name of the secret that you created earlier in Step 2.

14. Below the **Get secret** node, click the plus (**Insert a new action...**) button.

15. In the **Search for an action or connector** box, enter `HTTP`, and then under **HTTP**, select **HTTP**.

16. On the **Parameters** tab, set the following node properties:

    * For **URI**, enter `<unstructured-api-url>/workflows/<workflow-id>/run`, and replace the following placeholders:

      * Replace `<unstructured-api-url>` with your Unstructured API's workflow operations value.
      * Replace `<workflow-id>` with the ID of your Unstructured workflow.

      The **URI** should now look similar to the following:

      ```text theme={null}
      https://platform.unstructuredapp.io/api/v1/workflows/11111111-1111-1111-1111-111111111111/run
      ```

    * For **Method**, select **POST**.

    * For **Headers**, enter the following:

      * `accept`, set to `application/json`
      * `unstructured-api-key`, set to your key vault value. To do this, type `/` and select **Insert dynamic content**.
        Under **Get secret**, select **body/value**.

17. In the top navigation bar, click the disk (**Save**) button.

18. Click **Back** to go to the flow's home page.

19. In the top navigation bar, if the power button shows **Turn on**, click it to change it to **Turn off** (which means that the flow is on and active).

## Step 4: Trigger the flow and view results

1. On a separate tab in your web browser, go to the SharePoint document library that corresponds to both your Unstructured workflow's
   SharePoint source connector and the Power Automate flow's **When a file is created or modified (properties only)** node.
2. Add or update a file in the document library.
3. On a separate tab in your web browser, in Unstructured Pipelines for your account, click **Jobs** on the sidebar.
4. In the list of jobs, click the newly running job for your workflow. It might take a minute or so for the new job to appear in the list.
5. After the job status shows **Finished**, go to your destination location to see the results.

## Step 5 (Optional): Turn off the flow

To stop the Power Automate flow from running automatically, go back to the web browser tab from earlier in Step 3. Then,
with the flow's home page still open, in the top navigation bar, click the power (**Turn off**) button
to change it to **Turn on** (which means that the flow is off and inactive).


# Streamlit in Snowflake
Source: https://docs.unstructured.io/examplecode/tools/snowflake-streamlit



[Streamlit](https://streamlit.io/) is an open-source Python framework for data scientists and AI/ML engineers to
deliver dynamic data apps with only a few lines of code.

[Streamlit in Snowflake](https://www.snowflake.com/en/product/features/streamlit-in-snowflake/) enables data scientists
and Python developers to combine Streamlit's component-rich, open-source Python library with the scale, performance and
security of the Snowflake platform. Streamlit Python scripts can define user interface (UI) components such as
filters, graphs, sliders, and more to interact with your data.

In this example, you use Snowsight in your Snowflake account to create a simple Streamlit app that uses
[Snowflake Cortex Search for RAG](https://docs.snowflake.com/user-guide/snowflake-cortex/cortex-search/cortex-search-overview)
to ask natural-language questions about an existing table in your Snowflake account. This table
contains data that was generated by Unstructured. Answers are returned in natural-language,
chatbot-style format.

## Prerequisites

* A table in Snowflake that contains data that was generated by Unstructured. The
  target Snowflake table must have a column named `EMBEDDINGS` that will contains vector embeddings for the text in the table's `TEXT` column.
  The following Streamlit example app assumes that the `EMBEDDINGS` column contains 1,024 vector embeddings and has a data type of `VECTOR(FLOAT, 1024)`.

  To create this table, you can [create a custom Unstructured workflow](/pipelines/workflows#create-a-custom-workflow)
  that uses any supported [source connector](/pipelines/sources/overview) along with the
  [Snowflake destination connector](/pipelines/destinations/snowflake). Then
  [run](/pipelines/workflows#edit%2C-delete%2C-or-run-a-workflow) the workflow to
  generate the data and then insert that generated data into the target Snowflake table.

  After the data is inserted into the target Snowflake table, you can run the following Snowflake SQL statement to
  generate the 1,024 vector embeddings for the text in the table's `TEXT` column and then insert those generated vector
  embeddings into the table's `EMBEDDINGS` column. The model specified here for generating the vector embeddings is the
  same one that is used by the Streamlit example app:

  ```sql theme={null}
  UPDATE ELEMENTS
  SET EMBEDDINGS = SNOWFLAKE.CORTEX.EMBED_TEXT_1024(
      'snowflake-arctic-embed-l-v2.0', 
      TEXT
  );
  ```

  To learn how to run Snowflake SQL statements, see for example
  [Querying data using worksheets](https://docs.snowflake.com/user-guide/ui-snowsight-query).

* You must have the appropriate privileges to create and use a Streamlit app in your Snowflake account. These
  privileges include ones for the target table's parent database and schema as well as the Snowflake warehouse that
  runs the Streamlit app. For details, see
  [Getting started with Streamlit in Snowflake](https://docs.snowflake.com/developer-guide/streamlit/getting-started).

## Create and run the example app

<Steps>
  <Step title="Create the Streamlit app">
    1. In Snowsight for your Snowflake account, on the sidebar, click **Projects > Streamlit**.
    2. Click **+ Streamlit App**.
    3. For **App title**, enter a name for your app, such as `Unstructured Demo Streamlit App`.
    4. For **App location**, chose the target database and schema to store the app in.
    5. For **App warehouse**, choose the warehouse that you want to use to run your app and execute its queries.
    6. Click **Create**.
  </Step>

  <Step title="Add code to the Streamlit app">
    In this step, you add Python code to the Streamlit app that you created in the previous step.

    This step explains each part of the code as you add it. If you want to skip past these explanations, add the
    code in the [complete code example](#complete-code-example) all at once, and then skip ahead to
    the next step, "Run the Streamlit app."

    1. Import Python dependencies that get the current connection to the Snowflake database and schema and get Streamlit functions and features.

       ```python theme={null}
       from snowflake.snowpark.context import get_active_session
       import streamlit as st
       ```

    2. Get the current connection to the Snowflake database and schema.

       ```python theme={null}
       session = get_active_session() 
       ```

    3. Display the title of the app in the Streamlit UI, and get the user's search query from the Streamlit UI.

       ```python theme={null}
       st.title("Snowflake Cortex Search for RAG with Data from Unstructured")

       query = st.text_input("Enter your search query:")
       ```

    4. Get the user's search query and display a progress indicator in the UI.

       ```python theme={null}
       if query:
           with st.spinner("Embedding and retrieving..."):
       ```

    5. Use the user's search query to get the top result from the `ELEMENTS` table.
       The `ELEMENTS` table contains the data that was generated by Unstructured. The code uses the
       `SNOWFLAKE.CORTEX.EMBED_TEXT_1024` function to generate vector embeddings for the user's search query and the `VECTOR_COSINE_SIMILARITY`
       function to get the similarity between the vector embeddings for the user's search query and the vector embeddings for the `TEXT` column
       for each rown in the `ELEMENTS` table. The code then orders the results by similarity and limits the results to the row with the greatest similarity
       between the search query and the target text.

       ```python theme={null}
               top_result_df = session.sql(f"""
                   WITH query_embedding AS (
                       SELECT SNOWFLAKE.CORTEX.EMBED_TEXT_1024(
                           'snowflake-arctic-embed-l-v2.0', '{query}'
                       ) AS EMBED
                   )
                   SELECT 
                       e.TEXT,
                       VECTOR_COSINE_SIMILARITY(e.EMBEDDINGS, q.EMBED) AS similarity
                   FROM ELEMENTS e, query_embedding q
                   ORDER BY similarity DESC
                   LIMIT 1
               """).to_pandas()
       ```

    6. Get the `TEXT` column from the top result and use it as context for the user's search query.

       ```python theme={null}
               context = top_result_df["TEXT"][0]
       ```

    7. Use the user's search query and the context from the top result to get a response from Snowflake Cortex Search for RAG.
       The code uses the `SNOWFLAKE.CORTEX.COMPLETE` function to generate a response to the user's search query based on the context from the top result.

       ```python theme={null}
               completion_df = session.sql(f"""
                   SELECT SNOWFLAKE.CORTEX.COMPLETE(
                       'snowflake-arctic',
                   CONCAT('Context: ', $$ {context} $$, ' \\n\\nQuestion: {query}\\nAnswer:')
               ) AS RESPONSE
               """).to_pandas()
       ```

    8. Display the generated response in the Streamlit UI.

       ```python theme={null}
               st.write("Answer:")
               st.write(completion_df["RESPONSE"][0])
       ```
  </Step>

  <Step title="Run the Streamlit app">
    1. In the upper right corner, click **Run**.
    2. For **Enter your search query**, enter some natural-language question about the `TEXT` column in the table.
    3. Press **Enter**.

    Snowflake Cortex Search for RAG returns its answer to your question in natural-language, chatbot-style format.
  </Step>
</Steps>

## Complete code example

The full code example for the Streamlit app is as follows:

```python theme={null}
from snowflake.snowpark.context import get_active_session
import streamlit as st

session = get_active_session()

st.title("Snowflake Cortex Search for RAG with Data from Unstructured")

query = st.text_input("Enter your search query:")

if query:
    with st.spinner("Embedding and retrieving..."):

        top_result_df = session.sql(f"""
            WITH query_embedding AS (
                SELECT SNOWFLAKE.CORTEX.EMBED_TEXT_1024(
                    'snowflake-arctic-embed-l-v2.0', '{query}'
                ) AS EMBED
            )
            SELECT 
                e.TEXT,
                VECTOR_COSINE_SIMILARITY(e.EMBEDDINGS, q.EMBED) AS similarity
            FROM ELEMENTS e, query_embedding q
            ORDER BY similarity DESC
            LIMIT 1
        """).to_pandas()

        context = top_result_df["TEXT"][0]

        completion_df = session.sql(f"""
            SELECT SNOWFLAKE.CORTEX.COMPLETE(
                'snowflake-arctic',
                CONCAT('Context: ', $$ {context} $$, ' \\n\\nQuestion: {query}\\nAnswer:')
            ) AS RESPONSE
        """).to_pandas()

        st.write("Answer:")
        st.write(completion_df["RESPONSE"][0])
```

## Additional resources

* [Streamlit in Snowflake documentation](https://docs.snowflake.com/developer-guide/streamlit/about-streamlit)
* [Create and deploy Streamlit apps using Snowsight](https://docs.snowflake.com/developer-guide/streamlit/create-streamlit-ui)
* [Snowflake Solutions Developer Center for Streamlit](https://www.snowflake.com/en/developers/solutions-center/?tags=technology%2Fstreamlit)
* [Streamlit documentation](https://docs.streamlit.io/)


# VectorShift
Source: https://docs.unstructured.io/examplecode/tools/vectorshift



[VectorShift](https://vectorshift.ai/) is an integrated framework of no-code, low-code, and out of the box generative AI solutions
to build AI search engines, assistants, chatbots, and automations.

VectorShift's platform allows you to design, prototype, build, deploy,
and manage generative AI workflows and automation across two interfaces: no-code and code SDK.
This hands-on demonstration uses the no-code interface to walk you through creating a VectorShift pipeline project. This project
enables you to use GPT-4o-mini to chat in real time with a PDF document that is processed by Unstructured and has its processed data stored in a
[Pinecone](https://www.pinecone.io/) vector database.

## Prerequisites

<iframe title="YouTube video player" />

* A Pinecone account. [Get an account](https://app.pinecone.io/).

* A Pinecone API key in your Pinecone account. To create an API key, do the following:

  1. [Sign in to your Pinecone account and open the API keys page](https://app.pinecone.io/organizations/-/projects/-/keys).
  2. Click **+ API key**.
  3. For **API key name**, enter some descriptive name for the API key.
  4. Click **Create key**.
  5. Copy the generated API key to some secure location. You will not be able to access this API key again after you close the dialog.

* A Pinecone serverless index in your Pinecone account.

  Creating a serverless index on your own is optional.
  An index is not required to exist in advance.

  When you set up the connector, at runtime, the index behavior is as follows:

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * Your workflow must contain an embedder node, and the embedder node must specify the embedding model that Unstructured will use to generate the embeedings.
  * If an existing index name is specified,
    and the number of dimensions that Unstructured generates does not match the number of dimensions that are specified in the existing index's embedding settings, the run will fail.
    You must change the number of dimensions in your workflow's embedder node or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The
    new index's name will be `u<short-workflow-id>-<short-embedding-model-name>-<number-of-dimensions>`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * If an existing index name is specified, and Unstructured generates embeddings,
    but the number of dimensions that are generated does not match the existing index's embedding settings, the run will fail.
    You must change your Unstructured embedding settings or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The new index's name will be `unstructuredautocreated`.

  <Note>
    If you create a new index or use an existing one, Unstructured recommends that all records in the target index have a field
    named `record_id` with a string data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the index or, in some cases, the operation could fail altogether.
  </Note>

  To create a serverless index on your own, do the following:

  1. [Sign in to your Pinecone account and open the Create a new index page](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless).

  2. For **Enter index name**, enter some descriptive name for the index.

  3. For **Configuration**, select the check box labelled **Custom settings**, or click the tile labelled **Manual configuration**.

     <Warning>
       Do not click any of the other tiles, such as **text-embedding-3-large**. Clicking any of these other tiles will cause Pinecone to generate embeddings instead of
       having Unstructured generate them. If Pinecone generates embeddings instead of Unstructured, this could cause any related Unstructured workflows to fail.
     </Warning>

  4. For **Vector type**, select **Dense**.

  5. For **Dimension**, enter the number of dimensions for the embeddings that Unstructured will generate.

     <Warning>
       The number of dimensions that you enter here must match the number of dimensions for the embedding model that you use in any related Unstructured workflows or ingestion pipelines. If these numbers do not
       match in both places, this could cause any related Unstructured workflows or ingestion pipelines to fail.
     </Warning>

  6. For **Metric**, select **cosine**.

  7. Leave **Capacity mode** set to **Serverless**.

  8. You can leave **Cloud provider** and **Region** set to their default values, or you can select a cloud provider and region that is closest to you, if available.

  9. Click **Create index**.

* Within a Pinecone serverless index, custom [namespaces](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces) are supported but are not required.

Also:

* [Sign up for an OpenAI account](https://platform.openai.com/signup), and [get your OpenAI API key](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key).
* [Sign up for a VectorShift Starter account](https://app.vectorshift.ai/api/signup).
* If you do not already have an Unstructured account, sign up for one:

  1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
     After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
     </Note>

  2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

     <Note>
       For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
       or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
     </Note>

  3. Get your Unstructured API key:<br />

     a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

     <Note>
       For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
       for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
     </Note>

     b. Click **Generate API Key**.<br />
     c. Follow the on-screen instructions to finish generating the key.<br />
     d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

## Create and run the demonstration project

<Steps>
  <Step title="Get source data into Pinecone">
    Although you can use any [supported file type](/pipelines/supported-file-types) or data in any
    [supported source type](/pipelines/sources/overview) for the input into Pinecone, this demonstration uses [the text of the United States Constitution in PDF format](https://constitutioncenter.org/media/files/constitution.pdf).

    1. If you are not already signed in, sign in to your Unstructured account.
    2. [Create a source connector](/pipelines/sources/overview), if you do not already have one, to connect Unstructured to the source location where the PDF file is stored.
    3. [Create a Pinecone destination connector](/pipelines/destinations/pinecone), if you do not already have one, to connect Unstructured to your Pinecone serverless index.
    4. [Create a workflow](/pipelines/workflows#create-a-workflow) that references this source connector and destination connector.
    5. [Run the workflow](/pipelines/workflows#edit-delete-or-run-a-workflow).
  </Step>

  <Step title="Create the VectorShift project">
    1. Sign in to your VectorShift account dashboard.
    2. On the sidebar, click **Pipelines**.
    3. Click **New**.
    4. Click **Create Pipeline from Scratch**.

       <img alt="Create the VectorShift project" />
  </Step>

  <Step title="Add the Input node">
    In this step, you add a node to the pipeline. This node takes user-supplied chat messages and sends them as input to Pinecone, and as input to a text-based LLM, for contextual searching.

    In the top pipeline node chooser bar, on the **General** tab, click **Input**.

    <img alt="Adding the Input node" />
  </Step>

  <Step title="Add the Pinecone node">
    In this step, you add a node that connects to the Pinecone serverless index.

    1. In the top pipeline node chooser bar, on the **Integrations** tab, click **Pinecone**.
    2. In the **Pinecone** node, for **Embedding Model**, select **openai/text-embedding-3-large**.
    3. Click **Connected Account**.
    4. In the **Select Pinecone Account** dialog, click **Connect New**.
    5. Enter the **API Key** and **Region** for your Pinecone serverless index, and then click **Save**.
    6. For **Index**, selet the name of your Pinecone serverless index.
    7. Connect the **input\_1** output from the **Input** node to the **query** input in the **Pinecone** node.

       To make the connection, click and hold your mouse pointer inside of the circle next to **input\_1** in the **Input** node.
       While holding your mouse pointer, drag it over into the circle next to **query** in the **Pinecone** node. Then
       release your mouse pointer. A line appears between these two circles.

       <img alt="Adding the Pinecone node" />
  </Step>

  <Step title="Add the OpenAI LLM node">
    In this step, you add a node that builds a prompt and then sends it to a text-based LLM.

    1. In the top pipeline node chooser bar, on the **LLMs** tab, click **OpenAI**.

    2. In the **OpenAI LLM** node, for **System**, enter the following text:

       ```
       Answer the Question based on Context. Use Memory when relevant.
       ```

       <Tip>
         To answer the question, the preceding prompt uses the context along with general information that the text-based LLM is
         trained on. To use only the context to answer the question, you can change the prompt, for example to something like this:

         ```text theme={null}
         Answer the Question based only on the Context. Do not use any other sources of
         information. If the context does not provide enough information to answer the 
         question, reply with 'I do not have enough context to answer the question.' 
         Use Memory when relevant.
         ```
       </Tip>

    3. For **Prompt**, enter the following text:

       ```
       Question: {{Question}}
       Context: {{Context}}
       Memory: {{Memory}}
       ```

    4. For **Model**, select **gpt-4o-mini**.

    5. Check the box titled **Use Personal API Key**.

    6. For **API Key**, enter your OpenAI API key.

    7. Connect the **input\_1** output from the **Input** node to the **Question** input in the **OpenAI LLM** node.

    8. Connect the **output** output from the **Pinecone** node to the **Context** input in the **OpenAI LLM** node.

       <img alt="Adding the OpenAI LLM node" />
  </Step>

  <Step title="Add the Chat Memory node">
    In this step, you add a node that adds chat memory to the session.

    1. In the top pipeline node chooser bar, on the **Chat** tab, click **Chat Memory**.
    2. Connect the output from the **Chat Memory** node to the **Memory** input in the **OpenAI LLM** node.

       <img alt="Adding the Chat Memory node" />
  </Step>

  <Step title="Add the Output node">
    In this step, you add a node that displays the chat output.

    1. In the top pipeline node chooser bar, on the **General** tab, click **Output**.
    2. Connect the **response** output from the **OpenAI LLM** node to the input in the **Output** node.

       <img alt="Adding the Output node" />
  </Step>

  <Step title="Run the project">
    1. In the upper corner of the pipeline designer, click the play (**Run Pipeline**) button.

       <img alt="Running the pipeline" />

    2. In the chat pane, on the **Chatbot** tab, enter a question into the **Message Assistant** box, for example, `What rights does the fifth amendment guarantee?` Then press the send button.

       <img alt="Chatbot results" />

    3. Wait until the answer appears.

    4. Ask as many additional questions as you want to.
  </Step>
</Steps>

## Learn more

See the [VectorShift documentation](https://docs.vectorshift.ai/).


# Frequently asked questions
Source: https://docs.unstructured.io/faq/faq



View our [frequently asked questions](https://unstructured.io/pricing#faq).


# Chunking
Source: https://docs.unstructured.io/open-source/best-practices/chunking





# Embedding
Source: https://docs.unstructured.io/open-source/best-practices/embedding





# Document elements and metadata
Source: https://docs.unstructured.io/open-source/concepts/document-elements



`unstructured` simplifies and streamline the preprocessing of structured and unstructured documents for downstream tasks.
What that means is no matter where your data is and no matter what format that data is in, Unstructured's toolkit will
transform and preprocess that data into an easily digestible and usable format that is uniform across data formats.

When you partition a document with Unstructured, the result is a list of document `Element` objects.
These element objects represent different components of the source document.

## Element example

Here's an example of what an element might look like:

```
{'type': 'NarrativeText',
 'element_id': '5ef1d1117721f0472c1ad825991d7d37',
 'text': 'The Unstructured API documentation covers the following API services:',
 'metadata': {'last_modified': '2024-05-01T14:15:22',
  'page_number': 1,
  'languages': ['eng'],
  'parent_id': '56f24319ae258b735cac3ec2a271b1d9',
  'file_directory': '/content',
  'filename': 'Unstructured - Unstructured.html',
  'filetype': 'text/html'}}
```

As you can see, every element will have a `type`, an `element_id`, the extracted `text`, and some `metadata` which may
vary depending on the element type, document structure, and some additional parameters used during partitioning and/or chunking.

Let's explore some of these document element components in more detail.

## Element type

Instead of treating all documents like a wall of plain text, Unstructured preserves the semantic structure of the documents.
This gives you more control and flexibility over how you further use the processed documents and allows you to take their
structure into consideration. At the same time, normalizing data from various file formats to the Unstructured element
type scheme lets you treat all documents the same in your downstream processing, regardless of source format.
For example, if you plan to summarize a document, you may only be interested in the narrative
of the document, and not care about footers and headers. You can easily filter out the elements you don't need using their type.

Here are some examples of the element types your document may contain:

| Element type        | Description                                                                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Formula`           | An element containing formulas in a document.                                                                                                      |
| `FigureCaption`     | An element for capturing text associated with figure captions.                                                                                     |
| `NarrativeText`     | NarrativeText is an element consisting of multiple, well-formulated sentences. This excludes elements such titles, headers, footers, and captions. |
| `ListItem`          | ListItem is a NarrativeText element that is part of a list.                                                                                        |
| `Title`             | A text element for capturing titles.                                                                                                               |
| `Address`           | A text element for capturing physical addresses.                                                                                                   |
| `EmailAddress`      | A text element for capturing email addresses.                                                                                                      |
| `Image`             | A text element for capturing image metadata.                                                                                                       |
| `PageBreak`         | An element for capturing page breaks.                                                                                                              |
| `Table`             | An element for capturing tables.                                                                                                                   |
| `Header`            | An element for capturing document headers.                                                                                                         |
| `Footer`            | An element for capturing document footers.                                                                                                         |
| `CodeSnippet`       | An element for capturing code snippets.                                                                                                            |
| `PageNumber`        | An element for capturing page numbers.                                                                                                             |
| `UncategorizedText` | Base element for capturing free text from within document.                                                                                         |

If you apply chunking during partitioning of a document or later, you will also see the `CompositeElement` type.
`CompositeElement` is a chunk formed from text (non-Table) elements. It is only produced by chunking.
A composite element may be formed by combining one or more sequential elements produced by partitioning. For example,
several individual list items may be combined into a single chunk.

## Element ID

By default, the element ID is a SHA-256 hash of the element's text, its position on the page, page number it's on,
and the name of the document file - this is to ensure that the ID is deterministic and unique at the document level.
To obtain globally unique IDs in the output (UUIDs), you can pass `unique_element_ids=True` into any of the partition
functions. This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

## Metadata

Unstructured tracks a variety of metadata about the elements extracted from documents.
Here are a couple of examples of what element metadata enables you to do:

* filter document elements based on an element metadata value. For instance, you may want to limit your scope to elements from a certain page, or you may want to use only elements that have an email matching a regular expression in their metadata.
* map an element to the document page where it occurred so that original page can be retrieved when that element matches search criteria.

Metadata is tracked at the element level. You can access the metadata for a given document element with
`element.metadata`. For a dictionary representation, use `element.metadata.to_dict()`.

### Common metadata fields

All document types return the following metadata fields when the information is available from the source file:

| Metadata field name        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename`                 | Filename                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `file_directory`           | File directory                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `last_modified`            | Last modified Date                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `filetype`                 | File type                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `coordinates`              | XY Bounding Box Coordinates. See notes below for further details about the bounding box.                                                                                                                                                                                                                                                                                                                                                                                           |
| `parent_id`                | Element Hierarchy. `parent_id` may be used to infer where an element resides within the overall hierarchy of a document. For instance, a NarrativeText element may have a Title element as a parent (a “sub-title”), which in turn may have another Title element as its parent (a "title").                                                                                                                                                                                       |
| `category_depth`           | Element depth relative to other elements of the same category.  Category depth is the depth of an element relative to other elements of the same category. It’s set by a document partitioner and enables the hierarchy post-processor to compute more accurate hierarchies. Category depth may be set using native document hierarchies, e.g. reflecting \<H1>, \<H2>, or \<H3> tags within an HTML document or the indentation level of a bulleted list item in a Word document. |
| `image_base64`             | A Base64 representation of the detected image or table. Only applicable to image and table elements when High Res partitioning is used. After chunking, `image_base64` is not preserved in the output.                                                                                                                                                                                                                                                                             |
| `image_mime_type`          | MIME type of the image. Only applicable to image elements.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `text_as_html`             | HTML representation of extracted tables. Only applicable to table elements.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `languages`                | Document Languages.  At document level or element level. List is ordered by probability of being the primary language of the text.                                                                                                                                                                                                                                                                                                                                                 |
| `emphasized_text_contents` | Emphasized text (bold or italic) in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `emphasized_text_tags`     | Tags on text that is emphasized in the original document.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `orig_elements`            | For chunked elements, a list of the original elements that were used to create the current chunked element.                                                                                                                                                                                                                                                                                                                                                                        |
| `is_continuation`          | True if element is a continuation of a previous element.  Only relevant for chunking, if an element was divided into two due to max\_characters.                                                                                                                                                                                                                                                                                                                                   |
| `detection_class_prob`     | Detection model class probabilities.  From unstructured-inference, hi-res strategy.                                                                                                                                                                                                                                                                                                                                                                                                |

Notes on common metadata fields:

#### Metadata for document hierarchy

`parent_id` and `category_depth` enhance hierarchy detection to identify the document
structure in various file formats by measuring relative depth of an element within its category. This is especially
useful in documents with native hierarchies like HTML or Word files, where elements like headings or list items inherently define structure.

#### Element's coordinates

Some document types support location data for the elements, usually in the form of bounding boxes.
If it exists, an element's location data is available with `element.metadata.coordinates`. The `coordinates` property of an `ElementMetadata` stores:

* `points` : These specify the corners of the bounding box starting from the top left corner and proceeding counter-clockwise. The points represent pixels, the origin is in the top left and the `y` coordinate increases in the downward direction.
* `system`: The points have an associated coordinate system. A typical example of a coordinate system is `PixelSpace`, which is used for representing the coordinates of images. The coordinate system has a name, orientation, layout width, and layout height.

The Unstructured Open Source library offers a way to change the coordinates of an element to a new coordinate system by
using the `Element.convert_coordinates_to_new_system` method. If the `in_place` flag is `True`, the coordinate system
and points of the element are updated in place and the new coordinates are returned. If the `in_place` flag is `False`,
only the altered coordinates are returned.

```python theme={null}
from unstructured.documents.elements import Element
from unstructured.documents.coordinates import PixelSpace, RelativeCoordinateSystem

coordinates = ((10, 10), (10, 100), (200, 100), (200, 10))
coordinate_system = PixelSpace(width=850, height=1100)

element = Element(coordinates=coordinates, coordinate_system=coordinate_system)
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

element.convert_coordinates_to_new_system(RelativeCoordinateSystem(), in_place=True)
# Should now be in terms of new coordinate system
print(element.metadata.coordinates.to_dict())
print(element.metadata.coordinates.system.orientation)
print(element.metadata.coordinates.system.width)
print(element.metadata.coordinates.system.height)

```

### Additional metadata fields by document type

| Field name             | Applicable file types | Description                                                                                                                                                          |
| ---------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `attached_to_filename` | MSG                   | The name of the file that the attached file is attached to.                                                                                                          |
| `bcc_recipient`        | EML                   | The related [email](#email) BCC recipient.                                                                                                                           |
| `cc_recipient`         | EML                   | The related [email](#email) CC recipient.                                                                                                                            |
| `email_message_id`     | EML                   | The related [email](#email) message ID.                                                                                                                              |
| `header_footer_type`   | Word Doc              | The pages that a header or footer applies to in a [Word document](#microsoft-word-files): `primary`, `even_only`, and `first_page`.                                  |
| `image_path`           | PDF                   | The path to the image. This is useful when you want to extract the image and save it in a specified path instead of serializing the image within the processed data. |
| `image_mime_type`      | PDF                   | The MIME type of the image.                                                                                                                                          |
| `image_url`            | HTML                  | The URL to the image.                                                                                                                                                |
| `link_start_indexes`   | HTML, PDF             | A list of the index locations within the extracted content where the `links` can be found.                                                                           |
| `link_texts`           | HTML                  | A list of text strings that are associated with the `link_urls`.                                                                                                     |
| `link_urls`            | HTML                  | A list of URLs within the extracted content.                                                                                                                         |
| `links`                | PDF                   | A list of links within the extracted content.                                                                                                                        |
| `page_name`            | XLSX                  | The related sheet's name in an [Excel file](#microsoft-excel-files).                                                                                                 |
| `page_number`          | DOCX, PDF, PPT, XLSX  | The related file's page number.                                                                                                                                      |
| `section`              | EPUB                  | The book section title corresponding to a table of contents.                                                                                                         |
| `sent_from`            | EML                   | The related [email](#email) sender.                                                                                                                                  |
| `sent_to`              | EML                   | The related [email](#email) recipient.                                                                                                                               |
| `signature`            | EML                   | The related [email](#email) signature.                                                                                                                               |
| `subject`              | EML                   | The related [email](#email) subject.                                                                                                                                 |

Notes on additional metadata by document type:

#### Email

For emails, metadata will contain the following fields, where available:

* `bcc_recipient`
* `cc_recipient`
* `email_message_id`
* `sent_from`
* `sent_to`
* `signature`
* `subject`

`sent_from` is a list of strings because the [RFC 822](https://www.rfc-editor.org/rfc/rfc822) spec for emails allows for multiple sent from email addresses.

#### Microsoft Excel documents

For Excel documents, `ElementMetadata` will contain a `page_name` element, which corresponds to the sheet name in the Excel
document.

#### Microsoft Word documents

Headers and footers in Word documents include a `header_footer_type` indicating which page a header or footer applies to.
Valid values are `"primary"`, `"even_only"`, and `"first_page"`.

### Table-specific metadata

For `Table` elements, the raw text of the table will be stored in the `text` attribute for the Element, and HTML representation
of the table will be available in the element metadata under `element.metadata.text_as_html`. By default,
Unstructured will automatically extract all tables for all doc types unless you set `skip_infer_table_types` parameter.

Here's an example of a table element. The `text` of the element will look like this:

```
Dataset Base Model1 Large Model Notes PubLayNet [38] F / M M Layouts of modern scientific documents PRImA [3] M - Layouts of scanned modern magazines and scientific reports Newspaper [17] F - Layouts of scanned US newspapers from the 20th century TableBank [18] F F Table region on modern scientific and business document HJDataset [31] F / M - Layouts of history Japanese documents
```

And the `text_as_html` metadata for the same element will look like this:

```py theme={null}
<table><thead><th>Dataset</th><th>| Base Model’</th><th>| Notes</th></thead><tr><td>PubLayNet</td><td>[38] F/M</td><td>Layouts of modern scientific documents</td></tr><tr><td>PRImA [3]</td><td>M</td><td>Layouts of scanned modern magazines and scientific reports</td></tr><tr><td>Newspaper</td><td>F</td><td>Layouts of scanned US newspapers from the 20th century</td></tr><tr><td>TableBank</td><td>F</td><td>Table region on modern scientific and business document</td></tr><tr><td>HJDataset [31]</td><td>F/M</td><td>Layouts of history Japanese documents</td></tr></table>
```

### Data connector metadata fields

Documents processed through source connectors include additional document metadata. These additional fields only ever
appear if the source document was processed by a connector.

#### Common data connector metadata fields

* Data Source metadata (on json output):
  * url
  * version
  * date created
  * date modified
  * date processed
  * record locator
* Record locator is specific to each connector

#### Additional metadata fields by connector type (via record locator)

| Source connector      | Additional metadata              |
| --------------------- | -------------------------------- |
| airtable              | base id, table id, view id       |
| azure (from fsspec)   | protocol, remote file path       |
| box (from fsspec)     | protocol, remote file path       |
| confluence            | url, page id                     |
| discord               | channel                          |
| dropbox (from fsspec) | protocol, remote file path       |
| elasticsearch         | url, index name, document id     |
| fsspec                | protocol, remote file path       |
| google drive          | drive id, file id                |
| gcs (from fsspec)     | protocol, remote file path       |
| jira                  | base url, issue key              |
| onedrive              | user pname, server relative path |
| outlook               | message id, user email           |
| s3 (from fsspec)      | protocol, remote file path       |
| sharepoint            | server path, site url            |
| wikipedia             | page title, age url              |


# Models
Source: https://docs.unstructured.io/open-source/concepts/models

Depending on your need, `Unstructured` provides OCR-based and Transformer-based models to detect elements in the documents. The models are useful to detect the complex layout in the documents and predict the element types.

**Basic usage:**

```python theme={null}
elements = partition(filename=filename,
                     strategy="hi_res",
                     hi_res_model_name="yolox")
```

<Note>
  * To use any model with the partition, set the `strategy` to `hi_res` as shown above.

  * To maintain the consistency between the `unstructured` and `unstructured-api` libraries, we are deprecating the `model_name` parameter. Please use `hi_res_model_name` parameter when specifying a model.
</Note>

The `hi_res_model_name` parameter supports the `yolox` and `detectron2_onnx` arguments.

## Using a Non-Default Model

`Unstructured` will download the model specified in `UNSTRUCTURED_HI_RES_MODEL_NAME` environment variable. If not defined, it will download the default model.

There are three ways you can use the non-default model as follows:

1. Store the model name in the environment variable

```python theme={null}
import os
from unstructured.partition.pdf import partition_pdf

os.environ["UNSTRUCTURED_HI_RES_MODEL_NAME"] = "yolox"

out_yolox = partition_pdf("example-docs/pdf/layout-parser-paper-fast.pdf", strategy="hi_res")

```

2. Pass the model name in the `partition` function.

```python theme={null}
filename = "example-docs/pdf/layout-parser-paper-fast.pdf"

elements = partition(filename=filename,
                     strategy="hi_res",
                     hi_res_model_name="yolox")

```

3. Use [unstructured-inference](https://github.com/Unstructured-IO/unstructured-inference) library.

```python theme={null}
from unstructured_inference.models.base import get_model
from unstructured_inference.inference.layout import DocumentLayout

model = get_model("yolox")
layout = DocumentLayout.from_file("sample-docs/layout-parser-paper.pdf", detection_model=model)

```


# Partitioning strategies
Source: https://docs.unstructured.io/open-source/concepts/partitioning-strategies



For certain document types, such as images and PDFs, for example, Unstructured products offer a variety of different
ways to preprocess them, controlled by the `strategy` parameter.

PDF documents, for example, vary in quality and complexity. In simple cases, traditional NLP extraction techniques may
be enough to extract all the text out of a document. In other cases, advanced image-to-text models are required
to process a PDF. You can think of the strategies as being "rule-based" workflows (thus they are "fast"), or
"model-based" workflows (slower workflow because it requires model inference, but you get "higher resolution", thus "hi\_res").
When choosing a partitioning strategy for your files, you have to be mindful of the quality/speed trade-off.
To give you an example, the `fast` strategy is roughly 100x faster than leading image-to-text models.

**Available options:**

* `auto` (default strategy): The "auto" strategy will choose the partitioning strategy based on document characteristics and the function kwargs.
* `fast`:  The "rule-based" strategy leverages traditional NLP extraction techniques to quickly pull all the text elements. "Fast" strategy is not recommended for image-based file types.
* `hi_res`: The "model-based" strategy identifies the layout of the document. The advantage of "hi\_res" is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements.
* `ocr_only`: Another "model-based" strategy that leverages Optical Character Recognition to extract text from the image-based files.

**These strategies are available on the following partition functions:**

| Document Type            | Partition Function | Strategies                             | Table Support | Options                                                                                      |
| ------------------------ | ------------------ | -------------------------------------- | ------------- | -------------------------------------------------------------------------------------------- |
| Images (.png/.jpg/.heic) | partition\_image   | “auto”, “hi\_res”, “ocr\_only”         | Yes           | Encoding; Include Page Breaks; Infer Table Structure; OCR Languages, Strategy                |
| PDFs (.pdf)              | partition\_pdf     | “auto”, “fast”, “hi\_res”, “ocr\_only” | Yes           | Encoding; Include Page Breaks; Infer Table Structure; Max Partition; OCR Languages, Strategy |


# Chunking
Source: https://docs.unstructured.io/open-source/core-functionality/chunking

Chunking functions in `unstructured` use metadata and document elements detected with `partition` functions to post-process elements into more useful "chunks" for uses cases such as retrieval-augmented generation (RAG).

## Chunking Basics

Chunking in `unstructured` differs from other chunking mechanisms you may be familiar with. Typical approaches start with the text extracted from the document and form chunks based on plain-text features, character sequences like `"\n\n"` or `"\n"` that might indicate a paragraph boundary or list-item boundary.

Because `unstructured` uses specific knowledge about each document format to partition the document into semantic units (document elements), we only need to resort to text-splitting when a single element exceeds the desired maximum chunk size. Except in that case, all chunks contain one or more whole elements, preserving the coherence of semantic units established during partitioning.

A few concepts about chunking are worth introducing before discussing the details.

* Chunking is performed on *document elements*. It is a separate step performed *after* partitioning, on the elements produced by partitioning. (Although it can be combined with partitioning in a single step.)

* In general, chunking *combines* consecutive elements to form chunks as large as possible without exceeding the maximum chunk size.

* A single element that by itself exceeds the maximum chunk size is divided into two or more chunks using text-splitting.

* Chunking produces a sequence of `CompositeElement`, `Table`, or `TableChunk` elements. Each “chunk” is an instance of one of these three types.

## Chunking Options

The following options are available to tune chunking behaviors. These are keyword arguments that can be used in a partitioning or chunking function call. All these options have defaults and need only be specified when a non-default setting is required. Specific chunking strategies (such as `by_title`) may have additional options.

* `max_characters: int (default=500)` - the hard maximum size for a chunk. No chunk will exceed this number of characters. A single element that by itself exceeds this size will be divided into two or more chunks using text-splitting.

* `new_after_n_chars: int (default=max_characters)` - the “soft” maximum size for a chunk. A chunk that already exceeds this number of characters will not be extended, even if the next element would fit without exceeding the specified hard maximum. This can be used in conjunction with `max_characters` to set a “preferred” size, like “I prefer chunks of around 1000 characters, but I’d rather have a chunk of 1500 (max\_characters) than resort to text-splitting”. This would be specified with `(..., max_characters=1500, new_after_n_chars=1000)`.

* `overlap: int (default=0)` - only when using text-splitting to break up an oversized chunk, include this number of characters from the end of the prior chunk as a prefix on the next. This can mitigate the effect of splitting the semantic unit represented by the oversized element at an arbitrary position based on text length.

* `overlap_all: bool (default=False)` - also apply overlap between “normal” chunks, not just when text-splitting to break up an oversized element. Because normal chunks are formed from whole elements that each have a clean semantic boundary, this option may “pollute” normal chunks. You’ll need to decide based on your use-case whether this option is right for you.

## Chunking

Chunking can be performed as part of partitioning or as a separate step after partitioning:

### Specifying a chunking strategy while partitioning

Chunking can be performed as part of partitioning by specifying a value for the `chunking_strategy` argument. The current options are `basic` and `by_title` (described below).

```python theme={null}
from unstructured.partition.html import partition_html

chunks = partition_html(url=url, chunking_strategy="basic")

```

### Calling a chunking function

Chunking can also be performed separately from partitioning by calling a chunking function directly. This may be convenient, for example, when tuning chunking parameters. Chunking is typically faster than partitioning, especially when OCR or inference is used, so a faster feedback loop is possible by doing these separately:

```python theme={null}
from unstructured.chunking.basic import chunk_elements
from unstructured.partition.html import partition_html

url = "https://understandingwar.org/backgrounder/russian-offensive-campaign-assessment-august-27-2023-0"
elements = partition_html(url=url)
chunks = chunk_elements(elements)

# -- OR --

from unstructured.chunking.title import chunk_by_title

chunks = chunk_by_title(elements)

for chunk in chunks:
    print(chunk)
    print("\n\n" + "-"*80)
    input()

```

## Chunking Strategies

There are currently two chunking strategies, *basic* and *by\_title*. The `by_title` strategy shares most behaviors with
the basic strategy so we'll describe the baseline strategy first:

### "basic" chunking strategy

* The basic strategy combines sequential elements to maximally fill each chunk while respecting both the specified `max_characters` (hard-max) and `new_after_n_chars` (soft-max) option values.

* A single element that by itself exceeds the hard-max is isolated (never combined with another element) and then divided into two or more chunks using text-splitting.

* A `Table` element is always isolated and never combined with another element. A `Table` can be oversized, like any other text element, and in that case is divided into two or more `TableChunk` elements using text-splitting.

* If specified, `overlap` is applied between chunks formed by splitting oversized elements and is also applied between other chunks when `overlap_all` is `True`.

### "by\_title" chunking strategy

The `by_title` chunking strategy preserves section boundaries and optionally page boundaries as well. “Preserving” here means that a single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one started, even if the next element would fit in the prior chunk.

In addition to the behaviors of the `basic` strategy above, the `by_title` strategy has the following behaviors:

* **Detect section headings.** A `Title` element is considered to start a new section. When a `Title` element is encountered, the prior chunk is closed and a new chunk started, even if the `Title` element would fit in the prior chunk.

* **Respect page boundaries.** Page boundaries can optionally also be respected using the `multipage_sections` argument. This defaults to `True` meaning that a page break does *not* start a new chunk. Setting this to `False` will separate elements that occur on different pages into distinct chunks.

* **Combine small sections.** In certain documents, partitioning may identify a list-item or other short paragraph as a `Title` element even though it does not serve as a section heading. This can produce chunks substantially smaller than desired. This behavior can be mitigated using the `combine_text_under_n_chars` argument. This defaults to the same value as `max_characters` such that sequential small sections are combined to maximally fill the chunking window. Setting this to `0` will disable section combining.

## Recovering Chunk Elements

In general, a chunk consolidates multiple document elements to maximally fill a chunk of the desired size. Information is naturally lost in this consolidation, for example which element a portion of the text came from and certain metadata like page-number and coordinates which cannot always be resolved to a single value.

The original elements combined to make a chunk can be accessed using the .metadata.orig\_elements field on the chunk:

```python theme={null}
>>> elements = [
...     Title("Lorem Ipsum"),
...     NarrativeText("Lorem ipsum dolor sit."),
... ]
>>> chunk = chunk_elements(elements)[0]
>>> print(chunk.text)
'Lorem Ipsum\n\nLorem ipsum dolor sit.'
>>> print(chunk.metadata.orig_elements)
[Title("Lorem Ipsum"), NarrativeText("Lorem ipsum dolor sit.")]

```

These elements will contain all their original metadata so can be used to access metadata that cannot reliably be consolidated, for example:

```python python theme={null}
>>> {e.metadata.page_number for e in chunk.metadata.orig_elements}
{2, 3}

```

```
>>> [e.metadata.coordinates for e in chunk.metadata.orig_elements]
[<CoordinatesMetadata ...>, <CoordinatesMetadata ...>, ...]

```

```
>>> [
    e.metadata.image_path
    for e in chunk.metadata.orig_elements
    if e.metadata.image_path is not None
]
['/tmp/lorem.jpg', '/tmp/ipsum.png']

```

During serialization, `.metadata.orig_elements` is compressed into Base64 gzipped format. To deserialize `.metadata.orig_elements`, you can use the `elements_from_base64_gzipped_json`. For example:

```python theme={null}
from unstructured.partition.auto import partition
from unstructured.staging.base import elements_from_base64_gzipped_json

elements = partition('local-ingest-source/fake-email.eml', chunking_strategy='basic', include_orig_elements=True)

print("Before:\n")

for element in elements:
    metadata = element.metadata.to_dict()
    print(f"Element ID: {element.id}")
    print(f"  Compressed orig_elements: {metadata["orig_elements"]}")
    print("\n")

# Output:
# -------
# Before:
#
# Element ID: 083776ca703b1925e5fef69bb2635f1f
#   Compressed orig_elements: eJztlM1uGyEUhV8Fzbq2GGbwDFZUZdNFpDaLyOomiix+Lg7tABbcaRtFfffCJFXSyiuvLbFA3HsOHD7B/XMDE3gIuHem2ZJGmLaDnmnBGTfajF0/aqm6zSi4gnEwzQfSeEBpJMrS/9xYN8HeuAQaY3qqFlPUclq5cICMqxznpKGqamOQHmqLld9hBV66aQ1+qtVJhsMsi6SU7xsIh+ZhWc2499E462A5HqOMrdoyNrt22NJ+225WlG8prR65xrAp+sXji0R8hJ/kLioXcgzkyqfX6fUcMqZZ45zArF38uGy2yDEu4tuIf/VXb/PrEPEfqY7+VTurb+UG6hF3JTb5VLM1v0sF4dfL8qPLpAxJsDYs4QlGMmcgNiYyB4dLKa9rFnw6Ljd1K1OS6H7ArvoUw/+BjZveMt5zxQSVgo2S8kGr1ggtKLOivwA7E9iNP8aEMiA5Rhcwb99j2Tmc4BQOBtyOYuwko5J3RnLRtszowYpOWj5ccJyL4y5mKK8nASnC9yg+u4w3CP4UDWWMVEpT3fZUmb63RvORgR3o2Ovy011onEnjq4sT4AsPNc1wGsjDH93mEjs=

print ("After:\n")

for element in elements:
    metadata = element.metadata.to_dict()
    print(f"Element ID: {element.id}")
    orig_elements = elements_from_base64_gzipped_json(metadata["orig_elements"])
    print(f"  Uncompressed orig_elements:")
    for orig_element in orig_elements:
        print(f"    {orig_element.category}: {orig_element.text}")
    print("\n")

# Output:
# -------
# After:
#
# Element ID: 083776ca703b1925e5fef69bb2635f1f
#   Uncompressed orig_elements:
#     NarrativeText: This is a test email to use for unit tests.
#     Title: Important points:
#     ListItem: Roses are red
#     ListItem: Violets are blue
```

If High Res partitioning is used, the `orig_elements` value of each chunked element will contain an `image_base64` value for each of the `Image` and `Table` elements. To access the original content for an `image_base64` field,
[decompress and then Base64-decode](/open-source/how-to/get-chunked-elements) the `orig_elements` value. Then
[Base64-decode](/open-source/how-to/extract-image-block-types) the resulting `image_base64` values.

## Learn more

<Icon icon="blog" />  [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)


# Cleaning
Source: https://docs.unstructured.io/open-source/core-functionality/cleaning

As part of data preparation for an NLP model, it’s common to need to clean up your data prior to passing it into the model. If there’s unwanted content in your output, for example, it could impact the quality of your NLP model. To help with this, the `unstructured` library includes cleaning functions to help users sanitize output before sending it to downstream applications.

Some cleaning functions apply automatically. In the example in the **Partition** section, the output `Philadelphia Eaglesâ\x80\x99 victory` automatically gets converted to `Philadelphia Eagles' victory` in `partition_html` using the `replace_unicode_quotes` cleaning function. You can see how that works in the code snippet below:

```python theme={null}
from unstructured.cleaners.core import replace_unicode_quotes

replace_unicode_quotes("Philadelphia Eaglesâ\x80\x99 victory")

```

Document elements in `unstructured` include an `apply` method that allow you to apply the text cleaning to the document element without instantiating a new element. The `apply` method expects a callable that takes a string as input and produces another string as output. In the example below, we invoke the `replace_unicode_quotes` cleaning function using the `apply` method.

```python theme={null}
from unstructured.documents.elements import Text

element = Text("Philadelphia Eaglesâ\x80\x99 victory")
element.apply(replace_unicode_quotes)
print(element)

```

Since a cleaning function is just a `str -> str` function, users can also easily include their own cleaning functions for custom data preparation tasks. In the example below, we remove citations from a section of text.

```python theme={null}
import re

remove_citations = lambda text: re.sub("\[\d{1,3}\]", "", text)

element = Text("[1] Geolocated combat footage has confirmed Russian gains in the Dvorichne area northwest of Svatove.")
element.apply(remove_citations)
print(element)

```

See below for a full list of cleaning functions in the `unstructured` library.

## `bytes_string_to_string`

Converts an output string that looks like a byte string to a string using the specified encoding. This happens sometimes in `partition_html` when there is a character like an emoji that isn’t expected by the HTML parser. In that case, the encoded bytes get processed.

Examples:

```python theme={null}
from unstructured.cleaners.core import bytes_string_to_string

text = "Hello ð\x9f\x98\x80"
# The output should be "Hello 😀"
bytes_string_to_string(text, encoding="utf-8")

```

```python theme={null}
from unstructured.cleaners.core import bytes_string_to_string
from unstructured.partition.html import partition_html

text = """\n<html charset="utf-8"><p>Hello 😀</p></html>"""
elements = partition_html(text=text)
elements[0].apply(bytes_string_to_string)
# The output should be "Hello 😀"
elements[0].text

```

For more information about the `bytes_string_to_string` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean`

Cleans a section of text with options including removing bullets, extra whitespace, dashes and trailing punctuation. Optionally, you can choose to lowercase the output.

Options:

* Applies `clean_bullets` if `bullets=True`.

* Applies `clean_extra_whitespace` if `extra_whitespace=True`.

* Applies `clean_dashes` if `dashes=True`.

* Applies `clean_trailing_punctuation` if `trailing_punctuation=True`.

* Lowercases the output if `lowercase=True`.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean

# Returns "an excellent point!"
clean("● An excellent point!", bullets=True, lowercase=True)

# Returns "ITEM 1A: RISK FACTORS"
clean("ITEM 1A:     RISK-FACTORS", extra_whitespace=True, dashes=True)

```

For more information about the `clean` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_bullets`

Removes bullets from the beginning of text. Bullets that do not appear at the beginning of the text are not removed.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_bullets

# Returns "An excellent point!"
clean_bullets("● An excellent point!")

# Returns "I love Morse Code! ●●●"
clean_bullets("I love Morse Code! ●●●")

```

For more information about the `clean_bullets` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_dashes`

Removes dashes from a section of text. Also handles special characters such as `\u2013`.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_dashes

# Returns "ITEM 1A: RISK FACTORS"
clean_dashes("ITEM 1A: RISK-FACTORS\u2013")

```

For more information about the `clean_dashes` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_non_ascii_chars`

Removes non-ascii characters from a string.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_non_ascii_chars

text = "\x88This text contains ®non-ascii characters!●"

# Returns "This text contains non-ascii characters!"
clean_non_ascii_chars(text)

```

For more information about the `clean_non_ascii_chars` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_ordered_bullets`

Remove alphanumeric bullets from the beginning of text up to three “sub-section” levels.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_ordered_bullets

# Returns "This is a very important point"
clean_ordered_bullets("1.1 This is a very important point")

# Returns "This is a very important point ●"
clean_ordered_bullets("a.b This is a very important point ●")

```

For more information about the `clean_ordered_bullets` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_postfix`

Removes the postfix from a string if they match a specified pattern.

Options:

* Ignores case if `ignore_case` is set to `True`. The default is `False`.

* Strips trailing whitespace is `strip` is set to `True`. The default is `True`.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_postfix

text = "The end! END"

# Returns "The end!"
clean_postfix(text, r"(END|STOP)", ignore_case=True)

```

For more information about the `clean_postfix` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_prefix`

Removes the prefix from a string if they match a specified pattern.

Options:

* Ignores case if `ignore_case` is set to `True`. The default is `False`.

* Strips leading whitespace is `strip` is set to `True`. The default is `True`.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_prefix

text = "SUMMARY: This is the best summary of all time!"

# Returns "This is the best summary of all time!"
clean_prefix(text, r"(SUMMARY|DESCRIPTION):", ignore_case=True)

```

For more information about the `clean_prefix` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `clean_trailing_punctuation`

Removes trailing punctuation from a section of text.

Examples:

```python theme={null}
from unstructured.cleaners.core import clean_trailing_punctuation

# Returns "ITEM 1A: RISK FACTORS"
clean_trailing_punctuation("ITEM 1A: RISK FACTORS.")

```

For more information about the `clean_trailing_punctuation` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `group_broken_paragraphs`

Groups together paragraphs that are broken up with line breaks for visual or formatting purposes. This is common in `.txt` files. By default, `group_broken_paragraphs` groups together lines split by `\n`. You can change that behavior with the `line_split` kwarg. The function considers `\n\n` to be a paragraph break by default. You can change that behavior with the `paragraph_split` kwarg.

Examples:

```python theme={null}
from unstructured.cleaners.core import group_broken_paragraphs

text = """The big brown fox
was walking down the lane.

At the end of the lane, the
fox met a bear."""

group_broken_paragraphs(text)

```

```python theme={null}
import re
from unstructured.cleaners.core import group_broken_paragraphs

para_split_re = re.compile(r"(\s*\n\s*){3}")

text = """The big brown fox

was walking down the lane.


At the end of the lane, the

fox met a bear."""

group_broken_paragraphs(text, paragraph_split=para_split_re)

```

For more information about the `group_broken_paragraphs` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `remove_punctuation`

Removes ASCII and unicode punctuation from a string.

Examples:

```python theme={null}
from unstructured.cleaners.core import remove_punctuation

# Returns "A lovely quote"
remove_punctuation("“A lovely quote!”")

```

For more information about the `remove_punctuation` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `replace_unicode_quotes`

Replaces unicode quote characters such as `\x91` in strings.

Examples:

```
from unstructured.cleaners.core import replace_unicode_quotes

# Returns "“A lovely quote!”"
replace_unicode_characters("\x93A lovely quote!\x94")

# Returns ""‘A lovely quote!’"
replace_unicode_characters("\x91A lovely quote!\x92")

```

For more information about the `replace_unicode_quotes` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `translate_text`

The `translate_text` cleaning functions translates text between languages. `translate_text` uses the [Helsinki NLP MT models](https://huggingface.co/Helsinki-NLP) from `transformers` for machine translation. Works for Russian, Chinese, Arabic, and many other languages.

Parameters:

* `text`: the input string to translate.

* `source_lang`: the two letter language code for the source language of the text. If `source_lang` is not specified, the language will be detected using `langdetect`.

* `target_lang`: the two letter language code for the target language for translation. Defaults to `"en"`.

Examples:

```python theme={null}
from unstructured.cleaners.translate import translate_text

# Output is "I'm a Berliner!"
translate_text("Ich bin ein Berliner!")

# Output is "I can also translate Russian!"
translate_text("Я тоже можно переводать русский язык!", "ru", "en")

```

For more information about the `translate_text` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/translate.py).


# Embedding
Source: https://docs.unstructured.io/open-source/core-functionality/embedding



The Unstructured open-source library does not offer built-in support for calling embedding providers to obtain embeddings for pieces of text.

Alternatively, the [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest)
offer built-in support for calling embedding providers as part of an ingest pipeline. [Learn how](/open-source/how-to/embedding).

Also, you can use common third-party tools and libraries to get embeddings for document elements' text within JSON files that are
produced by calling the Unstructured open-source library. For example, the following sample Python script:

1. Takes an Unstructured open-source library-generated JSON file as input.
2. Reads in the JSON file's contents as a JSON object.
3. Uses the [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
   model on Hugging Face to generate embeddings for each `text` field of each document element in the JSON file.
4. Adds the generated embeddings next to each corresponding `text` field in the original JSON.
5. Saves the results back to the original JSON file.

```python Python theme={null}
# Filename: embeddings.py
# pip install langchain sentence_transformers

import sys
import json
from langchain_huggingface.embeddings import HuggingFaceEmbeddings

if __name__ == "__main__":
    embeddings = HuggingFaceEmbeddings(
        model_name="sentence-transformers/all-MiniLM-L6-v2"
    )
    
    # Get the JSON file's path.
    if len(sys.argv) < 2:
        print("Error: Specify the path to the input JSON file.")
        print("For example, 'python embeddings.py myfile.json'")
        sys.exit(1)

    file_path = sys.argv[1]
    
    try:
        # Get the JSON file's contents.
        with open(file_path, 'r') as file:
            file_elements = json.load(file)

        # Process each element in the JSON file.
        for element in file_elements:
            # Get the element's "text" field.
            text = element["text"]
            # Generate the embeddings for that "text" field.
            query_result = embeddings.embed_query(text)
            # Add the embeddings to that element as an "embeddings" field.
            element["embeddings"] = query_result

        # Save the updated JSON back into the original file.
        with open(file_path, 'w') as file:
            json.dump(file_elements, file, indent=2)

        print(f"Done! Updated JSON saved to '{file_path}'.")

    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
    except IOError:
        print(f"Error: Unable to access file '{file_path}'.")
```

## Additional resources

For information about how to use Python scripts to call various embedding providers, see for example:

* [Amazon Bedrock](https://docs.aws.amazon.com/code-library/latest/ug/python_3_bedrock-runtime_code_examples.html#amazon_titan_text_embeddings)
* [Hugging Face](https://huggingface.co/blog/getting-started-with-embeddings)
* [OctoAI](https://octo.ai/blog/introducing-octoais-embedding-api-to-power-your-rag-needs/)
* [OpenAI](https://platform.openai.com/docs/guides/embeddings)
* [together.ai](https://docs.together.ai/docs/embeddings-overview)
* [Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings)
* [Voyage AI](https://docs.voyageai.com/docs/embeddings)


# Extracting
Source: https://docs.unstructured.io/open-source/core-functionality/extracting



## `extract_datetimetz`

Extracts the date, time, and timezone in the `Received` field(s) from an `.eml` file. `extract_datetimetz` takes in a string and returns a datetime.datetime object from the input string.

```python theme={null}
from unstructured.cleaners.extract import extract_datetimetz

text = """from ABC.DEF.local ([ba23::58b5:2236:45g2:88h2]) by
  \n ABC.DEF.local2 ([ba23::58b5:2236:45g2:88h2%25]) with mapi id\
  n 32.88.5467.123; Fri, 26 Mar 2021 11:04:09 +1200"""

# Returns datetime.datetime(2021, 3, 26, 11, 4, 9, tzinfo=datetime.timezone(datetime.timedelta(seconds=43200)))
extract_datetimetz(text)

```

For more information about the `extract_datetimetz` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_email_address`

Extracts email addresses from a string input and returns a list of all the email addresses in the input string.

```python theme={null}
from unstructured.cleaners.extract import extract_email_address

text = """Me me@email.com and You <You@email.com>
    ([ba23::58b5:2236:45g2:88h2]) (10.0.2.01)"""

# Returns "['me@email.com', 'you@email.com']"
extract_email_address(text)

```

For more information about the `extract_email_address` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_ip_address`

Extracts IPv4 and IPv6 IP addresses in the input string and returns a list of all IP address in input string.

```python theme={null}
from unstructured.cleaners.extract import extract_ip_address

text = """Me me@email.com and You <You@email.com>
  ([ba23::58b5:2236:45g2:88h2]) (10.0.2.01)"""

# Returns "['ba23::58b5:2236:45g2:88h2', '10.0.2.01']"
extract_ip_address(text)

```

For more information about the `extract_ip_address` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_ip_address_name`

Extracts the names of each IP address in the `Received` field(s) from an `.eml` file. `extract_ip_address_name` takes in a string and returns a list of all IP addresses in the input string.

```python theme={null}
from unstructured.cleaners.extract import extract_ip_address_name

text = """from ABC.DEF.local ([ba23::58b5:2236:45g2:88h2]) by
  \n ABC.DEF.local2 ([ba23::58b5:2236:45g2:88h2%25]) with mapi id\
  n 32.88.5467.123; Fri, 26 Mar 2021 11:04:09 +1200"""

# Returns "['ABC.DEF.local', 'ABC.DEF.local2']"
extract_ip_address_name(text)

```

For more information about the `extract_ip_address_name` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_mapi_id`

Extracts the `mapi id` in the `Received` field(s) from an `.eml` file. `extract_mapi_id` takes in a string and returns a list of a string containing the `mapi id` in the input string.

```python theme={null}
from unstructured.cleaners.extract import extract_mapi_id

text = """from ABC.DEF.local ([ba23::58b5:2236:45g2:88h2]) by
  \n ABC.DEF.local2 ([ba23::58b5:2236:45g2:88h2%25]) with mapi id\
  n 32.88.5467.123; Fri, 26 Mar 2021 11:04:09 +1200"""

# Returns "['32.88.5467.123']"
extract_mapi_id(text)

```

For more information about the `extract_mapi_id` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_ordered_bullets`

Extracts alphanumeric bullets from the beginning of text up to three “sub-section” levels.

Examples:

```python theme={null}
from unstructured.cleaners.extract import extract_ordered_bullets

# Returns ("1", "1", None)
extract_ordered_bullets("1.1 This is a very important point")

# Returns ("a", "1", None)
extract_ordered_bullets("a.1 This is a very important point")

```

For more information about the `extract_ordered_bullets` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_text_after`

Extracts text that occurs after the specified pattern.

Options:

* If `index` is set, extract after the `(index + 1)`th occurrence of the pattern. The default is `0`.

* Strips trailing whitespace if `strip` is set to `True`. The default is `True`.

Examples:

```python theme={null}
from unstructured.cleaners.extract import extract_text_after

text = "SPEAKER 1: Look at me, I'm flying!"

# Returns "Look at me, I'm flying!"
extract_text_after(text, r"SPEAKER \d{1}:")

```

For more information about the `extract_text_after` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_text_before`

Extracts text that occurs before the specified pattern.

Options:

* If `index` is set, extract before the `(index + 1)`th occurrence of the pattern. The default is `0`.

* Strips leading whitespace if `strip` is set to `True`. The default is `True`.

Examples:

```python theme={null}
from unstructured.cleaners.extract import extract_text_before

text = "Here I am! STOP Look at me! STOP I'm flying! STOP"

# Returns "Here I am!"
extract_text_before(text, r"STOP")

```

For more information about the `extract_text_before` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `extract_us_phone_number`

Extracts a phone number from a section of text.

Examples:

```python theme={null}
from unstructured.cleaners.extract import extract_us_phone_number

# Returns "215-867-5309"
extract_us_phone_number("Phone number: 215-867-5309")

```

For more information about the `extract_us_phone_number` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/extract.py).

## `group_broken_paragraphs`

Groups together paragraphs that are broken up with line breaks for visual or formatting purposes. This is common in `.txt` files. By default, `group_broken_paragraphs` groups together lines split by `\n`. You can change that behavior with the `line_split` kwarg. The function considers `\n\n` to be a paragraph break by default. You can change that behavior with the `paragraph_split` kwarg.

Examples:

```python theme={null}
from unstructured.cleaners.core import group_broken_paragraphs

text = """The big brown fox
was walking down the lane.

At the end of the lane, the
fox met a bear."""

group_broken_paragraphs(text)

```

```python theme={null}
import re
from unstructured.cleaners.core import group_broken_paragraphs

para_split_re = re.compile(r"(\s*\n\s*){3}")

text = """The big brown fox

was walking down the lane.


At the end of the lane, the

fox met a bear."""

group_broken_paragraphs(text, paragraph_split=para_split_re)

```

For more information about the `group_broken_paragraphs` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `remove_punctuation`

Removes ASCII and unicode punctuation from a string.

Examples:

```python theme={null}
from unstructured.cleaners.core import remove_punctuation

# Returns "A lovely quote"
remove_punctuation("“A lovely quote!”")

```

For more information about the `remove_punctuation` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `replace_unicode_quotes`

Replaces unicode quote characters such as `\x91` in strings.

Examples:

```python theme={null}
from unstructured.cleaners.core import replace_unicode_quotes

# Returns "“A lovely quote!”"
replace_unicode_characters("\x93A lovely quote!\x94")

# Returns ""‘A lovely quote!’"
replace_unicode_characters("\x91A lovely quote!\x92")

```

For more information about the `replace_unicode_quotes` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/core.py).

## `translate_text`

The `translate_text` cleaning function translates text between languages. `translate_text` uses the [Helsinki NLP MT models](https://huggingface.co/Helsinki-NLP) from `transformers` for machine translation. Works for Russian, Chinese, Arabic, and many other languages.

Parameters:

* `text`: the input string to translate.

* `source_lang`: the two letter language code for the source language of the text. If `source_lang` is not specified, the language will be detected using `langdetect`.

* `target_lang`: the two letter language code for the target language for translation. Defaults to `"en"`.

Examples:

```python theme={null}
from unstructured.cleaners.translate import translate_text

# Output is "I'm a Berliner!"
translate_text("Ich bin ein Berliner!")

# Output is "I can also translate Russian!"
translate_text("Я тоже можно переводать русский язык!", "ru", "en")

```

For more information about the `translate_text` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/cleaners/translate.py).


# Core Functionality
Source: https://docs.unstructured.io/open-source/core-functionality/overview

The `unstructured` library includes functions to partition, chunk, clean, and stage raw source documents. These functions serve as the primary public interfaces within the library.

After reading this section, you should understand the following:

* How to partition a document into json or csv.

* How to remove unwanted content from document elements using cleaning functions.

* How to extract content from a document using the extraction functions.

* How to prepare data for downstream use cases using staging functions

* How to chunk partitioned documents for use cases such as retrieval-augmented generation (RAG).


# Partitioning
Source: https://docs.unstructured.io/open-source/core-functionality/partitioning

Partitioning functions in `unstructured` allow users to extract structured content from a raw unstructured document. These functions break a document down into elements such as `Title`, `NarrativeText`, and `ListItem`, enabling users to decide what content they’d like to keep for their particular application. If you’re training a summarization model, for example, you may only be interested in `NarrativeText`.

The easiest way to partition documents in unstructured is to use the `partition` function. If you call the `partition` function, `unstructured` will use `libmagic` to automatically determine the file type and invoke the appropriate partition function. In cases where `libmagic` is not available, filetype detection will fall back to using the file extension.

The following table shows the document types the `unstructured` library currently supports. `partition` will recognize each of these document types and route the document to the appropriate partitioning function. If you already know your document type, you can use the partitioning function listed in the table directly.

| Document Type                                                             | Partition Function | Strategies                             | Table Support | Options                                                                                      |
| ------------------------------------------------------------------------- | ------------------ | -------------------------------------- | ------------- | -------------------------------------------------------------------------------------------- |
| CSV Files (.csv)                                                          | partition\_csv     | N/A                                    | Yes           | None                                                                                         |
| E-mails (.eml)                                                            | partition\_email   | N/A                                    | No            | Encoding; Include Headers; Max Partition; Process Attachments                                |
| E-mails (.msg)                                                            | partition\_msg     | N/A                                    | No            | Encoding; Max Partition; Process Attachments                                                 |
| EPubs (.epub)                                                             | partition\_epub    | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Excel Documents (.xlsx/.xls)                                              | partition\_xlsx    | N/A                                    | Yes           | None                                                                                         |
| HTML Pages (.html/.htm)                                                   | partition\_html    | N/A                                    | No            | Encoding; Include Page Breaks                                                                |
| Images (.png/.jpg/.jpeg/.tiff/.bmp/.heic)                                 | partition\_image   | “auto”, “hi\_res”, “ocr\_only”         | Yes           | Encoding; Include Page Breaks; Infer Table Structure; OCR Languages, Strategy                |
| Markdown (.md)                                                            | partition\_md      | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Org Mode (.org)                                                           | partition\_org     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Open Office Documents (.odt)                                              | partition\_odt     | N/A                                    | Yes           | None                                                                                         |
| PDFs (.pdf)                                                               | partition\_pdf     | “auto”, “fast”, “hi\_res”, “ocr\_only” | Yes           | Encoding; Include Page Breaks; Infer Table Structure; Max Partition; OCR Languages, Strategy |
| Plain Text (.txt/.text/.log)                                              | partition\_text    | N/A                                    | No            | Encoding; Max Partition; Paragraph Grouper                                                   |
| PowerPoints (.ppt)                                                        | partition\_ppt     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| PowerPoints (.pptx)                                                       | partition\_pptx    | N/A                                    | Yes           | Include Page Breaks                                                                          |
| ReStructured Text (.rst)                                                  | partition\_rst     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Rich Text Files (.rtf)                                                    | partition\_rtf     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| TSV Files (.tsv)                                                          | partition\_tsv     | N/A                                    | Yes           | None                                                                                         |
| Word Documents (.doc)                                                     | partition\_doc     | N/A                                    | Yes           | Include Page Breaks                                                                          |
| Word Documents (.docx)                                                    | partition\_docx    | N/A                                    | Yes           | Include Page Breaks                                                                          |
| XML Documents (.xml)                                                      | partition\_xml     | N/A                                    | No            | Encoding; Max Partition; XML Keep Tags                                                       |
| Code Files (.js/.py/.java/ .cpp/.cc/.cxx/.c/.cs/ .php/.rb/.swift/.ts/.go) | partition\_text    | N/A                                    | No            | Encoding; Max Partition; Paragraph Grouper                                                   |

As shown in the examples below, the `partition` function accepts both filenames and file-like objects as input. `partition` also has some optional kwargs. For example, if you set `include_page_breaks=True`, the output will include `PageBreak` elements if the filetype supports it. Additionally you can bypass the filetype detection logic with the optional `content_type` argument which may be specified with either the `filename` or file-like object, `file`. You can find a full listing of optional kwargs in the documentation below.

```python theme={null}
from unstructured.partition.auto import partition


filename = os.path.join(EXAMPLE_DOCS_DIRECTORY, "layout-parser-paper-fast.pdf")
elements = partition(filename=filename, content_type="application/pdf")
print("\n\n".join([str(el) for el in elements][:10]))

```

```python theme={null}
from unstructured.partition.auto import partition


filename = os.path.join(EXAMPLE_DOCS_DIRECTORY, "layout-parser-paper-fast.pdf")
with open(filename, "rb") as f:
  elements = partition(file=f, include_page_breaks=True)
print("\n\n".join([str(el) for el in elements][5:15]))

```

The `unstructured` library also includes partitioning functions targeted at specific document types. The `partition` function uses these document-specific partitioning functions under the hood. There are a few reasons you may want to use a document-specific partitioning function instead of `partition`:

* If you already know the document type, filetype detection is unnecessary. Using the document-specific function directly, or passing in the `content_type` will make your program run faster.

* Fewer dependencies. You don’t need to install `libmagic` for filetype detection if you’re only using document-specific functions.

* Additional features. The API for partition is the least common denominator for all document types. Certain document-specific function include extra features that you may want to take advantage of. For example, `partition_html` allows you to pass in a URL so you don’t have to store the `.html` file locally. See the documentation below learn about the options available in each partitioning function.

Below we see an example of how to partition a document directly with the URL using the partition\_html function.

```python theme={null}
from unstructured.partition.html import partition_html

url = "https://www.cnn.com/2023/01/30/sport/empire-state-building-green-philadelphia-eagles-spt-intl/index.html"
elements = partition_html(url=url)
print("\n\n".join([str(el) for el in elements]))

```

## `partition`

The `partition` function is the simplest way to partition a document in `unstructured`. If you call the `partition` function, `unstructured` will attempt to detect the file type and route it to the appropriate partitioning function. All partitioning functions called within `partition` are called using the default kwargs. Use the document-type specific functions if you need to apply non-default settings. `partition` currently supports `.docx`, `.doc`, `.odt`, `.pptx`, `.ppt`, `.xlsx`, `.csv`, `.tsv`, `.eml`, `.msg`, `.rtf`, `.epub`, `.html`, `.xml`, `.pdf`, `.png`, `.jpg`, `.heic`, and `.txt` files. If you set the `include_page_breaks` kwarg to `True`, the output will include page breaks. This is only supported for `.pptx`, `.html`, `.pdf`, `.png`, `.heic`, and `.jpg`. The `strategy` kwarg controls the strategy for partitioning documents. Generally available strategies are “fast” for faster processing and “hi\_res” for more accurate processing.

```python theme={null}
import docx

from unstructured.partition.auto import partition

document = docx.Document()
document.add_paragraph("Important Analysis", style="Heading 1")
document.add_paragraph("Here is my first thought.", style="Body Text")
document.add_paragraph("Here is my second thought.", style="Normal")
document.save("mydoc.docx")

elements = partition(filename="mydoc.docx")

with open("mydoc.docx", "rb") as f:
    elements = partition(file=f)

```

```python theme={null}
from unstructured.partition.auto import partition

elements = partition(filename="example-docs/pdf/layout-parser-paper-fast.pdf")

```

The `partition` function also accepts a `url` kwarg for remotely hosted documents. If you want to force `partition` to treat the document as a particular MIME type, use the `content_type` kwarg in conjunction with `url`. Otherwise, `partition` will use the information from the `Content-Type` header in the HTTP response. The `ssl_verify` kwarg controls whether or not SSL verification is enabled for the HTTP request. By default it is on. Use `ssl_verify=False` to disable SSL verification in the request.

```python theme={null}
from unstructured.partition.auto import partition

url = "https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/LICENSE.md"
elements = partition(url=url)
elements = partition(url=url, content_type="text/markdown")

```

For more information about the `partition` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/auto.py).

## `partition_csv`

The `partition_csv` function pre-processes CSV files. The output is a single `Table` element. The `text_as_html` attribute in the element metadata will contain an HTML representation of the table.

Examples:

```python theme={null}
from unstructured.partition.csv import partition_csv

elements = partition_csv(filename="example-docs/stanley-cups.csv")
print(elements[0].metadata.text_as_html)

```

For more information about the `partition_csv` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/csv.py).

## `partition_doc`

The `partition_doc` partitioning function pre-processes Microsoft Word documents saved in the `.doc` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_doc` can take a filename or file-like object as input. `partition_doc` uses `libreoffice` to convert the file to `.docx` and then calls `partition_docx`. Ensure you have `libreoffice` installed before using `partition_doc`.

Examples:

```python theme={null}
from unstructured.partition.doc import partition_doc

elements = partition_doc(filename="example-docs/fake.doc")

```

For more information about the `partition_doc` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/doc.py).

## `partition_docx`

The `partition_docx` partitioning function pre-processes Microsoft Word documents saved in the `.docx` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_docx` can take a filename or file-like object as input, as shown in the two examples below.

Examples:

```python theme={null}
import docx

from unstructured.partition.docx import partition_docx

document = docx.Document()
document.add_paragraph("Important Analysis", style="Heading 1")
document.add_paragraph("Here is my first thought.", style="Body Text")
document.add_paragraph("Here is my second thought.", style="Normal")
document.save("mydoc.docx")

elements = partition_docx(filename="mydoc.docx")

with open("mydoc.docx", "rb") as f:
    elements = partition_docx(file=f)

```

In Word documents, headers and footers are specified per section. In the output, the `Header` elements will appear at the beginning of a section and `Footer` elements will appear at the end. MSFT Word headers and footers have a `header_footer_type` metadata field indicating where the header or footer applies. Valid values are `"primary"`, `"first_page"` and `"even_page"`.

`partition_docx` will include page numbers in the document metadata when page breaks are present in the document. The function will detect user inserted page breaks and page breaks inserted by the Word document renderer. Some (but not all) Word document renderers insert page breaks when you save the document. If your Word document renderer does not do that, you may not see page numbers in the output even if you see them visually when you open the document. If that is the case, you can try saving the document with a different renderer.

For more information about the `partition_docx` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/docx.py).

## `partition_email`

The `partition_email` function partitions `.eml` documents and works with exports from email clients such as Microsoft Outlook and Gmail. The `partition_email` takes a filename, file-like object, or raw text as input and produces a list of document `Element` objects as output. Also `content_source` can be set to `text/html` (default) or `text/plain` to process the html or plain text version of the email, respectively. In order for `partition_email` to return the header information as elements, `include_headers` must be set to `True`. Header information is captured in element metadata regardless of the `include_headers` setting.

| Header     | Element type | Metadata name      |
| ---------- | ------------ | ------------------ |
| Subject    | Subject      | subject            |
| From       | Sender       | sent\_from         |
| To         | Recipient    | sent\_to           |
| Cc         | Recipient    | cc\_recipient      |
| Bcc        | Recipient    | bcc\_recipient     |
| Received   | ReceivedInfo | NA                 |
| Message-ID | MetaData     | email\_message\_id |
| (other)    | MetaData     | NA                 |

Examples:

```python theme={null}
from unstructured.partition.email import partition_email

elements = partition_email(filename="example-docs/eml/fake-email.eml")

with open("example-docs/eml/fake-email.eml", "r") as f:
    elements = partition_email(file=f)

with open("example-docs/eml/fake-email.eml", "r") as f:
    text = f.read()
elements = partition_email(text=text)

with open("example-docs/eml/fake-email.eml", "r") as f:
    text = f.read()
elements = partition_email(text=text, content_source="text/plain")

with open("example-docs/eml/fake-email.eml", "r") as f:
    text = f.read()
elements = partition_email(text=text, include_headers=True)

```

`partition_email` includes a `max_partition` parameter that indicates the maximum character length for a document element. This parameter only applies if `"text/plain"` is selected as the `content_source`. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

You can optionally partition e-mail attachments by setting `process_attachments=True`. The following is an example of what the workflow looks like:

```python theme={null}
from unstructured.partition.email import partition_email

filename = "example-docs/eml/fake-email-attachment.eml"
elements = partition_email(filename=filename, process_attachments=True)
```

If the content of an email is PGP encrypted, `partition_email` will return an empty list of elements and emit a warning indicated the email is encrypted.

For more information about the `partition_email` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/email.py).

## `partition_epub`

The `partition_epub` function processes e-books in EPUB3 format. The function first converts the document to HTML using `pandocs` and then calls `partition_html`. You’ll need [pandocs](https://pandoc.org/installing.html) installed on your system to use `partition_epub`.

Examples:

```python theme={null}
from unstructured.partition.epub import partition_epub

elements = partition_epub(filename="example-docs/winter-sports.epub")

```

For more information about the `partition_epub` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/epub.py).

## `partition_html`

The `partition_html` function partitions an HTML document and returns a list of document `Element` objects. `partition_html` can take a filename, file-like object, string, or url as input.

The following three invocations of partition\_html() are essentially equivalent:

```python theme={null}
from unstructured.partition.html import partition_html

elements = partition_html(filename="example-docs/example-10k.html")

with open("example-docs/example-10k.html", "r") as f:
    elements = partition_html(file=f)

with open("example-docs/example-10k.html", "r") as f:
    text = f.read()
elements = partition_html(text=text)

```

The following illustrates fetching a url and partitioning the response content. The `ssl_verify` kwarg controls whether or not SSL verification is enabled for the HTTP request. By default it is on. Use `ssl_verify=False` to disable SSL verification in the request.

```python theme={null}
from unstructured.partition.html import partition_html

elements = partition_html(url="https://python.org/")

# you can also provide custom headers:

elements = partition_html(url="https://python.org/",
                          headers={"User-Agent": "YourScriptName/1.0 ..."})

# and turn off SSL verification

elements = partition_html(url="https://python.org/", ssl_verify=False)

```

For more information about the `partition_html` function, you can check the [source code](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/html/partition.py).

## `partition_image`

The `partition_image` function has the same API as `partition_pdf`. The only difference is that `partition_image` does not need to convert a PDF to an image prior to processing. The `partition_image` function supports `.png`, `.heic`, and `.jpg` files.

You can also specify what languages to use for OCR with the `languages` kwarg. For example, use `languages=["eng", "deu"]` to use the English and German language packs. See the [Tesseract documentation](https://github.com/tesseract-ocr/tessdata) for a full list of languages and install instructions.

Examples:

```python theme={null}
from unstructured.partition.image import partition_image

# Returns a List[Element] present in the pages of the parsed image document
elements = partition_image("example-docs/img/layout-parser-paper-fast.jpg")

# Applies the English and Swedish language pack for ocr
elements = partition_image("example-docs/img/layout-parser-paper-fast.jpg", languages=["eng", "swe"])

```

The `strategy` kwarg controls the method that will be used to process the PDF. The available strategies for images are `"auto"`, `"hi_res"` and `"ocr_only"`.

The `"auto"` strategy will choose the partitioning strategy based on document characteristics and the function kwargs. If `skip_infer_table_types` is set to an empty list, the strategy will be `"hi_res"` because that is the only strategy that currently extracts tables for PDFs. Otherwise, `"auto"` will choose `ocr_only`. `"auto"` is the default strategy.

The `"hi_res"` strategy will identify the layout of the document using `detectron2_onnx`. The advantage of “hi\_res” is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements. If `detectron2_onnx` is not available, the `"hi_res"` strategy will fall back to the `"ocr_only"` strategy.

The `"ocr_only"` strategy runs the document through Tesseract for OCR and then runs the raw text through `partition_text`. Currently, `"hi_res"` has difficulty ordering elements for documents with multiple columns. If you have a document with multiple columns that does not have extractable text, we recommend using the `"ocr_only"` strategy.

It is helpful to use `"ocr_only"` instead of `"hi_res"` if `detectron2_onnx` does not detect a text element in the image. To run example below, ensure you have the Korean language pack for Tesseract installed on your system.

```python theme={null}
from unstructured.partition.image import partition_image

filename = "example-docs/img/english-and-korean.png"
elements = partition_image(filename=filename, languages=["eng", "kor"], strategy="ocr_only")

```

For more information about the `partition_image` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/image.py).

## `partition_md`

The `partition_md` function provides the ability to parse markdown files. The following workflow shows how to use `partition_md`.

Examples:

```python theme={null}
from unstructured.partition.md import partition_md

elements = partition_md(filename="README.md")

```

For more information about the `partition_md` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/md.py).

## `partition_msg`

The `partition_msg` functions processes `.msg` files, which is a filetype specific to email exports from Microsoft Outlook. Email header information (`bcc_recipient`, `cc_recipient`, `email_message_id`, `sent_from`, `sent_to`, `subject`, etc.) is captured in element metadata.

Examples:

```python theme={null}
from unstructured.partition.msg import partition_msg

elements = partition_msg(filename="example-docs/fake-email.msg")

```

`partition_msg` includes a `max_partition` parameter that indicates the maximum character length for a document element. This parameter only applies if `"text/plain"` is selected as the `content_source`. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

You can optionally partition e-mail attachments by setting `process_attachments=True`. The following is an example of what the workflow looks like:

```python theme={null}
from unstructured.partition.msg import partition_msg

filename = "example-docs/fake-email-attachment.msg"
elements = partition_msg(filename=filename, process_attachments=True)
```

If the content of an email is PGP encrypted, `partition_msg` will return an empty list of elements and emit a warning indicated the email is encrypted.

For more information about the `partition_msg` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/msg.py).

## `partition_multiple_via_api`

`partition_multiple_via_api` is similar to `partition_via_api`, but allows you to partition multiple documents in a single REST API call. The result has the type `List[List[Element]]`, for example:

```python theme={null}
[
  [NarrativeText("Narrative!"), Title("Title!")],
  [NarrativeText("Narrative!"), Title("Title!")]
]

```

Examples:

```python theme={null}
from unstructured.partition.api import partition_multiple_via_api

filenames = ["example-docs/eml/fake-email.eml", "example-docs/fake.docx"]

documents = partition_multiple_via_api(filenames=filenames)

```

```python theme={null}
from contextlib import ExitStack

from unstructured.partition.api import partition_multiple_via_api

filenames = ["example-docs/eml/fake-email.eml", "example-docs/fake.docx"]
files = [open(filename, "rb") for filename in filenames]

with ExitStack() as stack:
    files = [stack.enter_context(open(filename, "rb")) for filename in filenames]
    documents = partition_multiple_via_api(files=files, metadata_filenames=filenames)

```

For more information about the `partition_multiple_via_api` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/api.py).

## `partition_odt`

The `partition_odt` partitioning function pre-processes Open Office documents saved in the `.odt` format. The function first converts the document to `.docx` using `pandoc` and then processes it using `partition_docx`.

Examples:

```python theme={null}
from unstructured.partition.odt import partition_odt

elements = partition_odt(filename="example-docs/fake.odt")

```

For more information about the `partition_odt` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/odt.py).

## `partition_org`

The `partition_org` function processes Org Mode (`.org`) documents. The function first converts the document to HTML using `pandoc` and then calls `partition_html`. You’ll need [pandoc](https://pandoc.org/installing.html) installed on your system to use `partition_org`.

Examples:

```python theme={null}
from unstructured.partition.org import partition_org

elements = partition_org(filename="example-docs/README.org")

```

For more information about the `partition_org` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/org.py).

## `partition_pdf`

The `partition_pdf` function segments a PDF document by using a document image analysis model. If you set `url=None`, the document image analysis model will execute locally. You need to install `unstructured[local-inference]` if you’d like to run inference locally. If you set the URL, `partition_pdf` will make a call to a remote inference server. `partition_pdf` also includes a `token` function that allows you to pass in an authentication token for a remote API call.

You can also specify what languages to use for OCR with the `languages` kwarg. For example, use `languages=["eng", "deu"]` to use the English and German language packs. See the [Tesseract documentation](https://github.com/tesseract-ocr/tessdata) for a full list of languages and install instructions. OCR is only applied if the text is not already available in the PDF document.

Examples:

```python theme={null}
from unstructured.partition.pdf import partition_pdf

# Returns a List[Element] present in the pages of the parsed pdf document
elements = partition_pdf("example-docs/pdf/layout-parser-paper-fast.pdf")

# Applies the English and Swedish language pack for ocr. OCR is only applied
# if the text is not available in the PDF.
elements = partition_pdf("example-docs/pdf/layout-parser-paper-fast.pdf", languages=["eng", "swe"])

```

The `strategy` kwarg controls the method that will be used to process the PDF. The available strategies for PDFs are `"auto"`, `"hi_res"`, `"ocr_only"`, and `"fast"`.

* The `"auto"` strategy will choose the partitioning strategy based on document characteristics and the function kwargs. If `skip_infer_table_types` is set to an empty list, the strategy will be `"hi_res"` because that is the only strategy that currently extracts tables for PDFs. Otherwise, `"auto"` will choose `"fast"` if the PDF text is extractable and `"ocr_only"` otherwise. `"auto"` is the default strategy.

* The `"hi_res"` strategy will identify the layout of the document using `detectron2_onnx`. The advantage of “hi\_res” is that it uses the document layout to gain additional information about document elements. We recommend using this strategy if your use case is highly sensitive to correct classifications for document elements. If `detectron2_onnx` is not available, the `"hi_res"` strategy will fall back to the `"ocr_only"` strategy.

* The `"ocr_only"` strategy runs the document through Tesseract for OCR and then runs the raw text through `partition_text`. Currently, `"hi_res"` has difficulty ordering elements for documents with multiple columns. If you have a document with multiple columns that does not have extractable text, we recommend using the `"ocr_only"` strategy. `"ocr_only"` falls back to `"fast"` if Tesseract is not available and the document has extractable text.

* The `"fast"` strategy will extract the text using `pdfminer` and process the raw text with `partition_text`. If the PDF text is not extractable, `partition_pdf` will fall back to `"ocr_only"`. We recommend using the `"fast"` strategy in most cases where the PDF has extractable text.

To extract images and elements as image blocks from a PDF, it is mandatory to set `strategy="hi_res"` when setting `extract_images_in_pdf=True`. With this configuration, detected images are saved in a specified directory or encoded within the file. However, keep in mind that `extract_images_in_pdf` is being phased out in favor of `extract_image_block_types`. This option allows you to specify types of images or elements, like “Image” or “Table”. If some extracted images have content clipped, you can adjust the padding by specifying two environment variables “EXTRACT\_IMAGE\_BLOCK\_CROP\_HORIZONTAL\_PAD” and “EXTRACT\_IMAGE\_BLOCK\_CROP\_VERTICAL\_PAD” (for example, EXTRACT\_IMAGE\_BLOCK\_CROP\_HORIZONTAL\_PAD = 20, EXTRACT\_IMAGE\_BLOCK\_CROP\_VERTICAL\_PAD = 10). For integrating these images directly into web applications or APIs, `extract_image_block_to_payload` can be used to convert them into `base64` format, including details about the image type, currently it’s always `image/jpeg`. Lastly, the `extract_image_block_output_dir` can be used to specify the filesystem path for saving the extracted images when not embedding them in payloads.

Examples:

```python theme={null}
from unstructured.partition.pdf import partition_pdf

partition_pdf(
    filename="path/to/your/pdf_file.pdf",                  # mandatory
    strategy="hi_res",                                     # mandatory to use ``hi_res`` strategy
    extract_images_in_pdf=True,                            # mandatory to set as ``True``
    extract_image_block_types=["Image", "Table"],          # optional
    extract_image_block_to_payload=False,                  # optional
    extract_image_block_output_dir="path/to/save/images",  # optional - only works when ``extract_image_block_to_payload=False``
    )

```

If a PDF is copy protected, `partition_pdf` can process the document with the `"hi_res"` strategy (which will treat it like an image), but cannot process the document with the `"fast"` strategy. If the user chooses `"fast"` on a copy protected PDF, `partition_pdf` will fall back to the `"hi_res"` strategy. If `detectron2_onnx` is not installed, `partition_pdf` will fail for copy protected PDFs because the document will not be processable by any of the available methods.

Examples:

```python theme={null}
from unstructured.partition.pdf import partition_pdf

# This will process without issue
elements = partition_pdf("example-docs/pdf/copy-protected.pdf", strategy="hi_res")

# This will output a warning and fall back to hi_res
elements = partition_pdf("example-docs/pdf/copy-protected.pdf", strategy="fast")

```

`partition_pdf` includes a `max_partition` parameter that indicates the maximum character length for a document element. This parameter only applies if the `"ocr_only"` strategy is used for partitioning. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

For more information about the `partition_pdf` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/pdf.py).

## `partition_ppt`

The `partition_ppt` partitioning function pre-processes Microsoft PowerPoint documents saved in the `.ppt` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_ppt` can take a filename or file-like object. `partition_ppt` uses `libreoffice` to convert the file to `.pptx` and then calls `partition_pptx`. Ensure you have `libreoffice` installed before using `partition_ppt`.

Examples:

```python theme={null}
from unstructured.partition.ppt import partition_ppt

elements = partition_ppt(filename="example-docs/fake-power-point.ppt")

```

For more information about the `partition_ppt` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/ppt.py).

## `partition_pptx`

The `partition_pptx` partitioning function pre-processes Microsoft PowerPoint documents saved in the `.pptx` format. This partition function uses a combination of the styling information in the document and the structure of the text to determine the type of a text element. The `partition_pptx` can take a filename or file-like object as input, as shown in the two examples below.

Examples:

```python theme={null}
from unstructured.partition.pptx import partition_pptx

elements = partition_pptx(filename="example-docs/fake-power-point.pptx")

with open("example-docs/fake-power-point.pptx", "rb") as f:
    elements = partition_pptx(file=f)

```

For more information about the `partition_pptx` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/pptx.py).

## `partition_rst`

The `partition_rst` function processes ReStructured Text (`.rst`) documents. The function first converts the document to HTML using `pandoc` and then calls `partition_html`. You’ll need [pandoc](https://pandoc.org/installing.html) installed on your system to use `partition_rst`.

Examples:

```
from unstructured.partition.rst import partition_rst

elements = partition_rst(filename="example-docs/README.rst")

```

For more information about the `partition_rst` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/rst.py).

## `partition_rtf`

The `partition_rtf` function processes rich text files. The function first converts the document to HTML using `pandocs` and then calls `partition_html`. You’ll need [pandocs](https://pandoc.org/installing.html) installed on your system to use `partition_rtf`.

Examples:

```python theme={null}
from unstructured.partition.rtf import partition_rtf

elements = partition_rtf(filename="example-docs/fake-doc.rtf")

```

For more information about the `partition_rtf` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/rtf.py).

## `partition_text`

The `partition_text` function partitions text files. The `partition_text` takes a filename, file-like object, and raw text as input and produces `Element` objects as output.

Examples:

```python theme={null}
from unstructured.partition.text import partition_text

elements = partition_text(filename="example-docs/fake-text.txt")

with open("example-docs/fake-text.txt", "r") as f:
  elements = partition_text(file=f)

with open("example-docs/fake-text.txt", "r") as f:
  text = f.read()
elements = partition_text(text=text)

```

If the text has extra line breaks for formatting purposes, you can group together the broken text using the `paragraph_grouper` kwarg. The `paragraph_grouper` kwarg is a function that accepts a string and returns another string.

Examples:

```python theme={null}
from unstructured.partition.text import partition_text
from unstructured.cleaners.core import group_broken_paragraphs


text = """The big brown fox
was walking down the lane.

At the end of the lane, the
fox met a bear."""

partition_text(text=text, paragraph_grouper=group_broken_paragraphs)

```

`partition_text` includes a `max_partition` parameter that indicates the maximum character length for a document element. The default value is `1500`, which roughly corresponds to the average character length for a paragraph. You can disable `max_partition` by setting it to `None`.

For more information about the `partition_text` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/text.py).

## `partition_tsv`

The `partition_tsv` function pre-processes TSV files. The output is a single `Table` element. The `text_as_html` attribute in the element metadata will contain an HTML representation of the table.

Examples:

```python theme={null}
from unstructured.partition.tsv import partition_tsv

elements = partition_tsv(filename="example-docs/stanley-cups.tsv")
print(elements[0].metadata.text_as_html)

```

For more information about the `partition_tsv` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/tsv.py).

## `partition_via_api`

`partition_via_api` allows users to partition documents using the hosted Unstructured API. The API partitions documents using the automatic `partition` function through the Unstructured SDK Client. This is helpful if you’re hosting the API yourself or running it locally through a container. You can pass in your API key using the `api_key` kwarg. You can use the `content_type` kwarg to pass in the MIME type for the file. If you do not explicitly pass it, the MIME type will be inferred.

```python theme={null}
from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(filename=filename, api_key="MY_API_KEY", content_type="message/rfc822")

with open(filename, "rb") as f:
  elements = partition_via_api(file=f, metadata_filename=filename, api_key="MY_API_KEY")

```

You can pass additional settings such as `strategy`, `languages` and `encoding` to the API through optional kwargs. These options get added to the request body when the API is called. See [the API documentation](https://api.unstructured.io/general/docs) for a full list of settings supported by the API.

```python theme={null}
from unstructured.partition.api import partition_via_api

filename = "example-docs/pdf/DA-1p.pdf"

elements = partition_via_api(
  filename=filename, api_key=api_key, strategy="auto"
)

```

If you are using the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview), you can use the `api_url` kwarg to point the `partition_via_api` function at your Unstructured Partition URL.

```python theme={null}
import os

from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(
  filename=filename,
  api_key=os.getenv("UNSTRUCTURED_API_KEY"),
  api_url=os.getenv("UNSTRUCTURED_API_URL")
)

```

If you are self-hosting or running the API locally, you can use the `api_url` kwarg to point the `partition_via_api` function at your self-hosted or local API. See [here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image) for documentation on how to run the API as a container locally.

```python theme={null}
import os

from unstructured.partition.api import partition_via_api

filename = "example-docs/eml/fake-email.eml"

elements = partition_via_api(
  filename=filename,
  api_key=os.getenv("UNSTRUCTURED_API_KEY"),
  api_url=os.getenv("UNSTRUCTURED_API_URL")
)

```

For more information about the `partition_via_api` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/api.py).

## `partition_xlsx`

The `partition_xlsx` function pre-processes Microsoft Excel documents. Each sheet in the Excel file will be stored as a `Table` object. The plain text of the sheet will be the `text` attribute of the `Table`. The `text_as_html` attribute in the element metadata will contain an HTML representation of the table.

Examples:

```python theme={null}
from unstructured.partition.xlsx import partition_xlsx

elements = partition_xlsx(filename="example-docs/stanley-cups.xlsx")
print(elements[0].metadata.text_as_html)

```

For more information about the `partition_xlsx` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/xlsx.py).

## `partition_xml`

The `partition_xml` function processes XML documents. If `xml_keep_tags=False`, the function only returns the text attributes from the tags. You can use `xml_path` in conjunction with `xml_keep_tags=False` to restrict the text extraction to specific tags. If `xml_keep_tags=True`, the function returns tag information in addition to tag text. `xml_keep_tags` is `False` be default.

```python theme={null}
from unstructured.partition.xml import partition_xml

elements = partition_xml(filename="example-docs/factbook.xml", xml_keep_tags=True)

elements = partition_xml(filename="example-docs/factbook.xml", xml_keep_tags=False)

```

For more information about the `partition_xml` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/xml.py).


# Staging
Source: https://docs.unstructured.io/open-source/core-functionality/staging



<Warning>
  Staging functions in the Unstructured open source library are being deprecated in favor of [destination connectors](/open-source/ingestion/destination-connectors/overview) in the [Unstructured Ingest CLI and Unstructured Ingest Python library](/open-source/ingestion/overview).
</Warning>

Staging functions in the `unstructured` package help prepare your data for ingestion into downstream systems. A staging function accepts a list of document elements as input and return an appropriately formatted dictionary as output. In the example below, we get our narrative text samples prepared for ingestion into LabelStudio using `the stage_for_label_studio` function. We can take this data and directly upload it into LabelStudio to quickly get started with an NLP labeling task.

```python theme={null}
import json
from unstructured.staging.label_studio import stage_for_label_studio

output = stage_for_label_studio(narrative_text)
print(json.dumps(output[:2], indent=4))

```

## `convert_to_csv`

Converts outputs to the initial structured data (ISD) format as a CSV string.

Examples:

```python theme={null}
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.base import convert_to_csv

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
isd_csv = convert_to_csv(elements)

```

For more information about the `convert_to_csv` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `convert_to_dataframe`

Converts a list of document `Element` objects to a `pandas` dataframe. The dataframe will have a `text` column with the text from the element and a `type` column indicating the element type, such as `NarrativeText` or `Title`.

Examples:

```python theme={null}
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.base import convert_to_dataframe

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
df = convert_to_dataframe(elements)

```

For more information about the `convert_to_dataframe` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `convert_to_dict`

Converts a list of `Element` objects to a dictionary. This is the default format for representing documents in `unstructured`.

Examples:

```python theme={null}
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.base import convert_to_dict

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
isd = convert_to_dict(elements)

```

For more information about the `convert_to_dict` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `dict_to_elements`

Converts a dictionary of the format produced by `convert_to_dict` back to a list of `Element` objects.

Examples:

```python theme={null}
from unstructured.staging.base import dict_to_elements

isd = [
  {"text": "My Title", "type": "Title"},
  {"text": "My Narrative", "type": "NarrativeText"}
]

# elements will look like:
# [ Title(text="My Title"), NarrativeText(text="My Narrative")]
elements = dict_to_elements(isd)

```

For more information about the `dict_to_elements` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/base.py).

## `stage_csv_for_prodigy`

Formats outputs in CSV format for use with [Prodigy](https://prodi.gy/docs/api-loaders). After running `stage_csv_for_prodigy`, you can write the results to a CSV file that is ready to be used with Prodigy.

Examples:

```python theme={null}
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.prodigy import stage_csv_for_prodigy

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"source": "news"}]
prodigy_csv_data = stage_csv_for_prodigy(elements, metadata)

# The resulting CSV file is ready to be used with Prodigy
with open("prodigy.csv", "w") as csv_file:
    csv_file.write(prodigy_csv_data)

```

For more information about the `stage_csv_for_prodigy` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/prodigy.py).

## `stage_for_argilla`

Convert a list of `Text` elements to an [Argilla Dataset](https://docs.argilla.io/en/latest/reference/python/python_client.html#python-ref-datasets). The type of Argilla dataset to be generated can be specified with `argilla_task` parameter. Valid values for `argilla_task` are `"text_classification"`, `"token_classification"`, and `"text2text"`. If `"token_classification"` is selected and `tokens` is not included in the optional kwargs, the `nltk` word tokenizer is used by default.

Examples:

```python theme={null}
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.argilla import stage_for_argilla

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"type": "text"}]

argilla_dataset = stage_for_argilla(elements, "text_classification", metadata=metadata)

```

For more information about the `stage_for_argilla` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/argilla.py).

## `stage_for_baseplate`-

The `stage_for_baseplate` staging function prepares a list of `Element` objects for ingestion into [Baseplate](https://docs.baseplate.ai/introduction), an LLM backend with a spreadsheet interface. After running the `stage_for_baseplate` function, you can use the [Baseplate API](https://docs.baseplate.ai/api-reference/documents/upsert-data-rows) to upload the documents to Baseplate. The following example code shows how to use the `stage_for_baseplate` function.

```python theme={null}
from unstructured.documents.elements import ElementMetadata, NarrativeText, Title
from unstructured.staging.baseplate import stage_for_baseplate

metadata = ElementMetadata(filename="fox.epub")

elements = [
  Title("A Wonderful Story About A Fox", metadata=metadata),
  NarrativeText(
    "A fox ran into the chicken coop and the chickens flew off!",
    metadata=metadata,
  ),
]

rows = stage_for_baseplate(elements)

```

The output will look like:

```json theme={null}
{
      "rows": [
          {
              "data": {
                  "element_id": "ad270eefd1cc68d15f4d3e51666d4dc8",
                  "text": "A Wonderful Story About A Fox",
                  "type": "Title",
              },
              "metadata": {"filename": "fox.epub"},
          },
          {
              "data": {
                  "element_id": "8275769fdd1804f9f2b55ad3c9b0ef1b",
                  "text": "A fox ran into the chicken coop and the chickens flew off!",
                  "type": "NarrativeText",
              },
              "metadata": {"filename": "fox.epub"},
          },
      ],
  }

```

For more information about the `stage_for_baseplate` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/baseplate.py).

## `stage_for_datasaur`

Formats a list of `Text` elements as input to token based tasks in Datasaur.

Example:

```python theme={null}
from unstructured.documents.elements import Text
from unstructured.staging.datasaur import stage_for_datasaur

elements  = [Text("Text1"),Text("Text2")]
datasaur_data = stage_for_datasaur(elements)

```

The output is a list of dictionaries, each one with two keys: “text” with the content of the element and “entities” with an empty list.

You can also specify entities in the `stage_for_datasaur` function. Entities you specify in the input will be included in the entities key in the output. The list of entities is a list of dictionaries and must have all of the keys in the example below. The list of entities must be the same length as the list of elements. Use an empty list for any elements that do not have any entities.

Example:

```python theme={null}
from unstructured.documents.elements import Text
from unstructured.staging.datasaur import stage_for_datasaur

elements  = [Text("Hi my name is Matt.")]
entities = [[{"text": "Matt", "type": "PER", "start_idx": 11, "end_idx": 15}]]
datasaur_data = stage_for_datasaur(elements, entities)

```

For more information about the `stage_for_datasaur` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/datasaur.py).

## `stage_for_label_box`

Formats outputs for use with [LabelBox](https://docs.labelbox.com/docs/overview). LabelBox accepts cloud-hosted data and does not support importing text directly. The `stage_for_label_box` does the following:

* Stages the data files in the `output_directory` specified in function arguments to be uploaded to a cloud storage service.

* Returns a config of type `List[Dict[str, Any]]` that can be written to a `json` file and imported into LabelBox.

**Note:** `stage_for_label_box` does not upload the data to remote storage such as S3. Users can upload the data to S3 using `aws s3 sync ${output_directory} ${url_prefix}` after running the `stage_for_label_box` staging function.

Examples:

The following example demonstrates generating a `config.json` file that can be used with LabelBox and uploading the staged data files to an S3 bucket.

```python theme={null}
import os
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.label_box import stage_for_label_box

# The S3 Bucket name where data files should be uploaded.
S3_BUCKET_NAME = "labelbox-staging-bucket"

# The S3 key prefix (I.e. directory) where data files should be stored.
S3_BUCKET_KEY_PREFIX = "data/"

# The URL prefix where the data files will be accessed.
S3_URL_PREFIX = f"https://{S3_BUCKET_NAME}.s3.amazonaws.com/{S3_BUCKET_KEY_PREFIX}"

# The local output directory where the data files will be staged for uploading to a Cloud Storage service.
LOCAL_OUTPUT_DIRECTORY = "/tmp/labelbox-staging"

elements = [Title(text="Title"), NarrativeText(text="Narrative")]

labelbox_config = stage_for_label_box(
    elements,
    output_directory=LOCAL_OUTPUT_DIRECTORY,
    url_prefix=S3_URL_PREFIX,
    external_ids=["id1", "id2"],
    attachments=[[{"type": "RAW_TEXT", "value": "Title description"}], [{"type": "RAW_TEXT", "value": "Narrative Description"}]],
    create_directory=True,
)

# The resulting JSON config file is ready to be used with LabelBox.
with open("config.json", "w+") as labelbox_config_file:
    json.dump(labelbox_config, labelbox_config_file, indent=4)


# Upload staged data files to S3 from local output directory.
def upload_staged_files():
    from s3fs import S3FileSystem
    fs = S3FileSystem()
    for filename in os.listdir(LOCAL_OUTPUT_DIRECTORY):
        filepath = os.path.join(LOCAL_OUTPUT_DIRECTORY, filename)
        upload_key = os.path.join(S3_BUCKET_KEY_PREFIX, filename)
        fs.put_file(lpath=filepath, rpath=os.path.join(S3_BUCKET_NAME, upload_key))

upload_staged_files()

```

For more information about the `stage_for_label_box` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/label_box.py).

## `stage_for_label_studio`

Formats outputs for upload to LabelStudio. After running `stage_for_label_studio`, you can write the results to a JSON folder that is ready to be included in a new LabelStudio project.

Examples:

```python theme={null}
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.label_studio import stage_for_label_studio

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
label_studio_data = stage_for_label_studio(elements, text_field="my_text", id_field="my_id")

# The resulting JSON file is ready to be uploaded to LabelStudio
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

You can also include pre-annotations and predictions as part of your LabelStudio upload.

The `annotations` kwarg is a list of lists. If `annotations` is specified, there must be a list of annotations for each element in the `elements` list. If an element does not have any annotations, use an empty list. The following shows an example of how to upload annotations for the “Text Classification” task in LabelStudio:

```python theme={null}
import json

from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_studio import (
    stage_for_label_studio,
    LabelStudioAnnotation,
    LabelStudioResult,
)



elements = [NarrativeText(text="Narrative")]
annotations = [[
  LabelStudioAnnotation(
      result=[
          LabelStudioResult(
              type="choices",
              value={"choices": ["Positive"]},
              from_name="sentiment",
              to_name="text",
          )
      ]
  )
]]
label_studio_data = stage_for_label_studio(
    elements,
    annotations=annotations,
    text_field="my_text",
    id_field="my_id"
)

# The resulting JSON file is ready to be uploaded to LabelStudio
# with annotations included
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

Similar to annotations, the `predictions` kwarg is also a list of lists. A `prediction` is an annotation with the addition of a `score` value. If `predictions` is specified, there must be a list of predictions for each element in the `elements` list. If an element does not have any predictions, use an empty list. The following shows an example of how to upload predictions for the “Text Classification” task in LabelStudio:

```python theme={null}
import json

from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_studio import (
    stage_for_label_studio,
    LabelStudioPrediction,
    LabelStudioResult,
)



elements = [NarrativeText(text="Narrative")]
predictions = [[
  LabelStudioPrediction(
      result=[
          LabelStudioResult(
              type="choices",
              value={"choices": ["Positive"]},
              from_name="sentiment",
              to_name="text",
          )
      ],
      score=0.68
  )
]]
label_studio_data = stage_for_label_studio(
    elements,
    predictions=predictions,
    text_field="my_text",
    id_field="my_id"
)

# The resulting JSON file is ready to be uploaded to LabelStudio
# with annotations included
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

The following shows an example of how to upload annotations for the “Named Entity Recognition” task in LabelStudio:

```python theme={null}
import json

from unstructured.documents.elements import NarrativeText
from unstructured.staging.label_studio import (
    stage_for_label_studio,
    LabelStudioAnnotation,
    LabelStudioResult,
)



elements = [NarrativeText(text="Narrative")]
annotations = [[
  LabelStudioAnnotation(
      result=[
          LabelStudioResult(
              type="labels",
              value={"start": 0, "end": 9, "text": "Narrative", "labels": ["MISC"]},
              from_name="label",
              to_name="text",
          )
      ]
  )
]]
label_studio_data = stage_for_label_studio(
    elements,
    annotations=annotations,
    text_field="my_text",
    id_field="my_id"
)

# The resulting JSON file is ready to be uploaded to LabelStudio
# with annotations included
with open("label_studio.json", "w") as f:
    json.dump(label_studio_data, f, indent=4)

```

See the [LabelStudio docs](https://labelstud.io/tags/labels.html) for a full list of options for labels and annotations.

For more information about the `stage_for_label_studio` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/label_studio.py).

## `stage_for_prodigy`

Formats outputs in JSON format for use with [Prodigy](https://prodi.gy/docs/api-loaders). After running `stage_for_prodigy`, you can write the results to a JSON file that is ready to be used with Prodigy.

Examples:

```python theme={null}
import json

from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.prodigy import stage_for_prodigy

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"type": "text"}]
prodigy_data = stage_for_prodigy(elements, metadata)

# The resulting JSON file is ready to be used with Prodigy
with open("prodigy.json", "w") as f:
    json.dump(prodigy_data, f, indent=4)

```

**Note**: Prodigy recommends `.jsonl` format for feeding data to API loaders. After running `stage_for_prodigy`, you can use the `save_as_jsonl` utility function to save the formatted data to a `.jsonl` file that is ready to be used with Prodigy.

```python theme={null}
from unstructured.documents.elements import Title, NarrativeText
from unstructured.staging.prodigy import stage_for_prodigy
from unstructured.utils import save_as_jsonl

elements = [Title(text="Title"), NarrativeText(text="Narrative")]
metadata = [{"type": "title"}, {"type": "text"}]
prodigy_data = stage_for_prodigy(elements, metadata)

# The resulting jsonl file is ready to be used with Prodigy.
save_as_jsonl(prodigy_data, "prodigy.jsonl")

```

For more information about the `stage_for_prodigy` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/prodigy.py).

## `stage_for_transformers`

Prepares `Text` elements for processing in `transformers` pipelines by splitting the elements into chunks that fit into the model’s attention window.

Examples:

```python theme={null}
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

from unstructured.documents.elements import NarrativeText
from unstructured.staging.huggingface import stage_for_transformers

model_name = "hf-internal-testing/tiny-bert-for-token-classification"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

nlp = pipeline("ner", model=model, tokenizer=tokenizer)

text = """From frost advisories this morning to a strong cold front expected later this week, the chance of fall showing up is real.

There's a refreshing crispness to the air, and it looks to get only more pronounced as the week goes on.

Frost advisories were in place this morning across portions of the Appalachians and coastal Maine as temperatures dropped into the 30s.

Temperatures this morning were in the 40s as far south as the Florida Panhandle.

And Maine even had a few reports of their first snow of the season Sunday. More cities could see their first snow later this week.

Yes, hello fall!

As temperatures moderate during the next few days, much of the east will stay right around seasonal norms, but the next blast of cold air will be strong and come with the potential for hazardous conditions.

"A more active fall weather pattern is expected to evolve by the end of this week and continuing into the weekend as a couple of cold fronts move across the central and eastern states," the Weather Prediction Center said.

The potent cold front will come in from Canada with a punch of chilly air, heavy rain and strong wind.

The Weather Prediction Center has a slight risk of excessive rainfall for much of the Northeast and New England on Thursday, including places like New York City, Buffalo and Burlington, so we will have to look out for flash flooding in these areas.

"More impactful weather continues to look likely with confidence growing that our region will experience the first real fall-like system with gusty to strong winds and a period of moderate to heavy rain along and ahead of a cold front passage," the National Weather Service office in Burlington wrote.

The potential for very heavy rain could accompany the front, bringing up to two inches of rain for much of the area, and isolated locations could see even more.

"Ensembles [forecast models] show median rainfall totals by Wednesday night around a half inch, with a potential for some spots to see around one inch, our first substantial rainfall in at least a couple of weeks," the weather service office in Grand Rapids noted, adding, "It may also get cold enough for some snow to mix in Thursday night to Friday morning, especially in the higher terrain north of Grand Rapids toward Cadillac."

There is also a chance for very strong winds to accompany the system.

The weather service is forecasting winds of 30-40 mph ahead of the cold front, which could cause some tree limbs to fall and sporadic power outages.

Behind the front, temperatures will fall.

"East Coast, with highs about 5-15 degrees below average to close out the workweek and going into next weekend, with highs only in the 40s and 50s from the Great Lakes to the Northeast on most days," the Weather Prediction Center explained.

By the weekend, a second cold front will drop down from Canada and bring a reinforcing shot of chilly air across the eastern half of the country."""

elements = stage_for_transformers([NarrativeText(text=text)], tokenizer)

```

The following optional keyword arguments can be specified in `stage_for_transformers`:

<Note>
  * `buffer`: Indicates the number of tokens to leave as a buffer for the attention window. This is to account for special tokens like `[CLS]` that can appear at the beginning or end of an input sequence.

  * `max_input_size`: The size of the attention window for the model. If not specified, the default is the `model_max_length` attribute on the tokenizer object.

  * `split_function`: The function used to split the text into chunks to consider for adding to the attention window. Splits on spaces be default.

  * `chunk_separator`: The string used to concat adjacent chunks when reconstructing the text. Uses spaces by default.

  If you need to operate on text directly instead of `unstructured` `Text` objects, use the `chunk_by_attention_window` helper function. Simply modify the example above to include the following:

  ```python theme={null}
  from unstructured.staging.huggingface import chunk_by_attention_window

  chunks = chunk_by_attention_window(text, tokenizer)

  results = [nlp(chunk) for chunk in chunks]

  ```
</Note>

For more information about the `stage_for_transformers` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/huggingface.py).

## `stage_for_weaviate`

The `stage_for_weaviate` staging function prepares a list of `Element` objects for ingestion into the [Weaviate](https://weaviate.io/) vector database. You can create a schema in Weaviate for the unstructured outputs using the following workflow:

```python theme={null}
from unstructured.staging.weaviate import create_unstructured_weaviate_class

import weaviate

# Change `class_name` if you want the class for unstructured documents in Weaviate
# to have a different name
unstructured_class = create_unstructured_weaviate_class(class_name="UnstructuredDocument")
schema = {"classes": [unstructured_class]}

client = weaviate.Client("http://localhost:8080")
client.schema.create(schema)

```

Once the schema is created, you can batch upload documents to Weaviate using the following workflow. See the [Weaviate documentation](https://weaviate.io/developers/weaviate) for more details on options for uploading data and querying data once it has been uploaded.

```python theme={null}
from unstructured.partition.pdf import partition_pdf
from unstructured.staging.weaviate import stage_for_weaviate

import weaviate
from weaviate.util import generate_uuid5


filename = "example-docs/pdf/layout-parser-paper-fast.pdf"
elements = partition_pdf(filename=filename, strategy="fast")
data_objects = stage_for_weaviate(elements)

client = weaviate.Client("http://localhost:8080")

with client.batch(batch_size=10) as batch:
    for data_object in tqdm.tqdm(data_objects):
        batch.add_data_object(
            data_object,
            unstructured_class_name,
            uuid=generate_uuid5(data_object),
        )

```

For more information about the `stage_for_weaviate` function, you can check the [source code here](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/staging/weaviate.py).


# Set embedding behavior
Source: https://docs.unstructured.io/open-source/how-to/embedding



<Note>
  The following information applies only to the [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest).

  The Unstructured SDKs for Python and JavaScript/TypeScript, and the Unstructured open-source library, do not support this functionality.
</Note>

## Concepts

You can use the [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) or the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest)
to generate *embeddings* after the partitioning and chunking steps in an ingest pipeline. The chunking step is particularly important to ensure that the text pieces
(also known as the *documents* or *elements*) can fit the input limits of an *embedding model*.

You generate embeddings by specifying an embedding model that is provided or used by an *embedding provider*.
An [embedding model](https://python.langchain.com/v0.2/docs/concepts/#embedding-models) creates arrays of numbers
known as *vectors*, representing the text that is extracted by Unstructured.
These vectors are stored or *embedded* next to the data itself.

These vector embeddings allow *vector databases* to more quickly and efficiently analyze and process these inherent
properties and relationships between data. For example, you can save the extracted text along with its embeddings in a *vector store*.
When a user queries a retrieval-augmented generation (RAG) application, the application can use a vector database to perform a similarity search in that vector store
and then return the documents whose embeddings are the closest to that user's query.

Learn more about [chunking](https://unstructured.io/blog/chunking-for-rag-best-practices) and
[embedding](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).

## Generate embeddings

<iframe title="YouTube video player" />

To use the Ingest CLI or Ingest Python library to generate embeddings, do the following:

1. Choose an embedding provider that you want to use from among the following allowed providers, and note the provider's ID:

   <Note>
     The following list assumes that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS)
     for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), you are limited to
     the provider and model names that are supported by the Unstructured API.
     [See the list of supported provider names](/api-reference/workflow/nodes/embedder/embedder).
   </Note>

   * The provider ID `bedrock` for [Amazon Bedrock](https://aws.amazon.com/bedrock/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/bedrock/).
   * `huggingface` for [Hugging Face](https://huggingface.co/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/huggingfacehub/).
   * `mixedbread-ai` for [Mixedbread](https://www.mixedbread.ai/). [Learn more](https://www.mixedbread.ai/docs/embeddings/overview).
   * `octoai` for [Octo AI](https://octo.ai/). [Learn more](https://octo.ai/docs/text-gen-solution/using-unstructured-io-for-embedding-documents).
   * `openai` for [OpenAI](https://openai.com/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/openai/).
   * `togetherai` for [Together.ai](https://www.together.ai/). [Learn more](https://docs.together.ai/docs/embedding-models).
   * `vertexai` for [Google Vertex AI PaLM](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/google_vertex_ai_palm/).
   * `voyageai` for [Voyage AI](https://www.voyageai.com/). [Learn more](https://python.langchain.com/v0.2/docs/integrations/text_embedding/voyageai/).

2. Run the following command to install the required Python package for the embedding provider:

   * For `bedrock`, run `pip install "unstructured-ingest[bedrock]"`.
   * For `huggingface`, run `pip install "unstructured-ingest[embed-huggingface]"`.
   * For `mixedbread-ai`, run `pip install "unstructured-ingest[embed-mixedbreadai]"`.
   * For `octoai`, run `pip install "unstructured-ingest[embed-octoai]"`.
   * For `openai`, run `pip install "unstructured-ingest[openai]"`.
   * For `togetherai`, run `pip install "unstructured-ingest[togetherai]"`.
   * For `vertexai`, run `pip install "unstructured-ingest[embed-vertexai]"`.
   * For `voyageai`, run `pip install "unstructured-ingest[embed-voyageai]"`.

3. For the following embedding providers, you can choose the model that you want to use. If you do choose a model, note the model's name:

   <Note>
     The following list assumes that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS)
     for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), you are limited to
     the model names that are supported by the Unstructured API.
     [See the list of supported model names](/api-reference/workflow/nodes/embedder/embedder).
   </Note>

   * `bedrock`. [Choose a model](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). No default model is provided. [Learn more about the supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html).
   * `huggingface`. [Choose a model](https://huggingface.co/models?other=embeddings), or use the default model [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).
   * `mixedbread-ai`. [Choose a model](https://www.mixedbread.ai/docs/embeddings/models), or use the default model [mixedbread-ai/mxbai-embed-large-v1](https://www.mixedbread.ai/docs/embeddings/mxbai-embed-large-v1).
   * `octoai`. [Choose a model](https://octo.ai/blog/supercharge-rag-performance-using-octoai-and-unstructured-embeddings/), or use the default model `thenlper/gte-large`.
   * `openai`. [Choose a model](https://platform.openai.com/docs/guides/embeddings/embedding-models), or use the default model `text-embedding-ada-002`.
   * `togetherai`. [Choose a model](https://docs.together.ai/docs/embedding-models), or use the default model `togethercomputer/m2-bert-80M-32k-retrieval`.
   * `vertexai`. [Choose a model](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api), or use the default model `text-embedding-05`.
   * `voyageai`.  [Choose a model](https://docs.voyageai.com/docs/embeddings). No default model is provided.

4. Note the special settings to connect to the provider:

   <Note>
     The following special settings assume that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS)
     for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), do not include any of these special
     settings. Unstructured uses its own internal special settings when using the specified provider to generate the embeddings.
   </Note>

   * For `bedrock`, you'll need an AWS access key value, the corresponding AWS secret access key value, and the corresponding AWS Region identifier. [Get an AWS access key and secret access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
   * For `huggingface`, if you use a gated model (a model with special conditions that you must accept before you can use it, or a privately published model), you'll need an HF inference API key value, beginning with `hf_`. [Get an HF inference API key](https://huggingface.co/docs/api-inference/en/quicktour#get-your-api-token). To learn whether your model requires an HF inference API key, see your model provider's documentation.
   * For `mixedbread-ai`, you'll need a Mixedbread API key value. [Get a Mixedbread API key](https://www.mixedbread.ai/dashboard?next=api-keys).
   * For `octoai`, you'll need an Octo AI API token value. [Get an Octo AI API token](https://octo.ai/docs/getting-started/how-to-create-octoai-access-token).
   * For `openai`, you'll need an OpenAI API key value. [Get an OpenAI API key](https://platform.openai.com/docs/quickstart/create-and-export-an-api-key).
   * For `togetherai`, you'll need a together.ai API key value. [Get a together.ai API key](https://docs.together.ai/reference/authentication-1).
   * For `vertexai`, you'll need the path to a Google Cloud credentials JSON file. Learn more [here](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) and [here](https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth).
   * For `voyageai`, you'll need a Voyage AI API key value. [Get a Voyage AI API key](https://docs.voyageai.com/docs/api-key-and-installation#authentication-with-api-keys).

5. Now, apply all of this information as follows, and then run your command or code:

<AccordionGroup>
  <Accordion title="Ingest CLI">
    <Note>
      The following options assume that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS)
      for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), do not include any of the following options:

      * `--embedding-api-key`
      * `--embedding-aws-access-key-id`
      * `--embedding-aws-secret-access-key`
      * `--embedding-aws-region`

      Unstructured uses its own internal settings for these options when using the specified provider to generate the embeddings.
    </Note>

    For the [source connector](/open-source/ingestion/source-connectors/overview) command:

    * Set the command's `--embedding-provider` to the provider's ID, for example `huggingface`.
    * Set `--embedding-model-name` to the model name, as applicable, for example `sentence-transformers/sentence-t5-xl`. Or omit this to use the default model, as applicable.
    * Set `--embedding-api-key` to the provider's required API key value or credentials JSON file path, as appropriate.
    * For `bedrock`:

      * Set `--embedding-aws-access-key-id` to the AWS access key value.
      * Set `--embedding-aws-secret-access-key` to the corresponding AWS secret access key value.
      * Set `--embedding-aws-region` to the corresponding AWS Region identifier.
  </Accordion>

  <Accordion title="Ingest Python library">
    <Note>
      The following parameters assume that you are calling the embedding provider directly. If you are calling Unstructured's software-as-a-service (SaaS)
      for processing instead (for example, by specifying an Unstructured API key and an Unstructured SaaS URL), do not include any of the following parameters:

      * `embedding_api_key`
      * `embedding_aws_access_key_id`
      * `embedding_aws_secret_access_key`
      * `embedding_aws_region`

      Unstructured uses its own internal settings for these parameters when using the specified provider to generate the embeddings.
    </Note>

    For the [source connector's](/open-source/ingestion/source-connectors/overview) `EmbedderConfig` object:

    * Set the `embedding_provider` parameter to the provider's ID, for example `huggingface`.
    * Set `embedding_model_name` to the model name, as applicable, for example `sentence-transformers/sentence-t5-xl`. Or omit this to use the default model, as applicable.
    * Set `embedding_api_key` to the provider's required API key value or credentials JSON file path, as appropriate.
    * For `bedrock`:

      * Set `embedding_aws_access_key_id` to the AWS access key value.
      * Set `embedding_aws_secret_access_key` to the corresponding AWS secret access key value.
      * Set `embedding_aws_region` to the corresponding AWS Region identifier.
  </Accordion>
</AccordionGroup>


# Examples
Source: https://docs.unstructured.io/open-source/how-to/examples

This page provides some examples of accessing Unstructured by using the Unstructured Ingest CLI and the Unstructured Ingest Python library.

These examples assume that you have already followed the instructured to set up the
[Unstructured Ingest CLI](/open-source/ingestion/ingest-cli) and the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest).

### Changing partition strategy for a PDF

Here's how you can modify partition strategy for a PDF file, and select an alternative model to use with Unstructured API.
The `hi_res` strategy supports different models, and the default is `layout_v1.1.0`.

<iframe title="YouTube video player" />

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --strategy hi_res \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                strategy="hi_res",
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

If you have a local deployment of the Unstructured API, you can use other supported models, such as `yolox`.

### Specifying the language of a document for better OCR results

For better OCR results, you can specify what languages your document is in using the `languages` parameter.
[View the list of available languages](https://github.com/tesseract-ocr/tessdata).

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --strategy ocr_only \
        --ocr-languages kor \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                strategy="ocr_only",
                ocr_languages=["kor"],
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

### Saving bounding box coordinates

When elements are extracted from PDFs or images, it may be useful to get their bounding boxes as well.
Set the `coordinates` parameter to `true` to add this field to the elements in the response.

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"coordinates\":\"true\", \"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "coordinates": True,
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

### Returning unique element IDs

By default, the element ID is a SHA-256 hash of the element text. This is to ensure that
the ID is deterministic. One downside is that the ID is not guaranteed to be unique.
Different elements with the same text will have the same ID, and there could also be hash collisions.
To use UUIDs in the output instead, set `unique_element_ids=true`. Note: this means that the element IDs
will be random, so with every partition of the same file, you will get different IDs.
This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"unique_element_ids\":\"true\", \"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "unique_element_ids": True,
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>

### Adding the chunking step after partitioning

You can combine partitioning and subsequent chunking in a single request by setting the `chunking_strategy` parameter.
By default, the `chunking_strategy` is set to `None`, and no chunking is performed.

[//]: # "TODO: add a link to the concepts section about chunking strategies. Need to create the shared Concepts section first"

<AccordionGroup>
  <Accordion title="Ingest CLI">
    ```bash CLI theme={null}
    unstructured-ingest \
      local \
        --input-path $LOCAL_FILE_INPUT_DIR \
        --output-dir $LOCAL_FILE_OUTPUT_DIR \
        --chunking-strategy by_title \
        --chunk-max-characters 1024 \
        --partition-by-api \
        --api-key $UNSTRUCTURED_API_KEY \
        --partition-endpoint $UNSTRUCTURED_API_URL \
        --strategy hi_res \
        --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
    ```
  </Accordion>

  <Accordion title="Ingest Python">
    ```python Python theme={null}
    import os

    from unstructured_ingest.pipeline.pipeline import Pipeline
    from unstructured_ingest.interfaces import ProcessorConfig
    from unstructured_ingest.processes.connectors.local import (
        LocalIndexerConfig,
        LocalDownloaderConfig,
        LocalConnectionConfig,
        LocalUploaderConfig
    )
    from unstructured_ingest.processes.partitioner import PartitionerConfig
    from unstructured_ingest.processes.chunker import ChunkerConfig

    if __name__ == "__main__":
        Pipeline.from_configs(
            context=ProcessorConfig(),
            indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
            downloader_config=LocalDownloaderConfig(),
            source_connection_config=LocalConnectionConfig(),
            partitioner_config=PartitionerConfig(
                partition_by_api=True,
                api_key=os.getenv("UNSTRUCTURED_API_KEY"),
                partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
                strategy="hi_res",
                additional_partition_args={
                    "split_pdf_page": True,
                    "split_pdf_allow_failed": True,
                    "split_pdf_concurrency_level": 15
                }
            ),
            chunker_config=ChunkerConfig(
                chunking_strategy="by_title",
                chunk_max_characters=1024
            ),
            uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
        ).run()
    ```
  </Accordion>
</AccordionGroup>


# Extract images and tables from documents
Source: https://docs.unstructured.io/open-source/how-to/extract-image-block-types



## Task

You want to get, decode, and show elements, such as images and tables, that are embedded in a PDF document.

## Approach

Extract the Base64-encoded representation of specific elements, such as images and tables, in the document.
For each of these extracted elements, decode the Base64-encoded representation of the element into its original visual representation
and then show it.

## To run this example

You will need a document that is one of the document types supported by the `extract_image_block_types` argument.
See the `extract_image_block_types` entry in [API Parameters](/api-reference/legacy-api/partition/api-parameters).
This example uses a PDF file with embedded images and tables.

## Code

For the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest), you can use the standard Python
[json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
file that the Ingest Python library outputs after the processing is complete.

```python Python theme={null}
import json, base64, io
from PIL import Image

def get_image_block_types(input_json_file_path: str):
    with open(input_json_file_path, 'r') as file:
        file_elements = json.load(file)

    for element in file_elements:
        if "image_base64" in element["metadata"]:
            # Decode the Base64-encoded representation of the 
            # processed "Image" or "Table" element into its original
            # visual representation, and then show it.
            image_data = base64.b64decode(element["metadata"]["image_base64"])
            image = Image.open(io.BytesIO(image_data))
            image.show()

if __name__ == "__main__":
    # Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/pdf/embedded-images-tables.pdf

    # Specify where to get the local file, relative to this .py file.
    get_image_block_types(
        input_json_file_path="local-ingest-output/embedded-images-tables.json"
    )
```


# Process a subset of files
Source: https://docs.unstructured.io/open-source/how-to/filter-files



<Note>
  The following information applies only to the [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest).

  The Unstructured SDKs for Python and JavaScript/TypeScript and the Unstructured open-source library do not support this functionality.
</Note>

## Task

You want to process only files with specified extensions, only files at or below a specified size, or both.

## Approach

For the Ingest CLI, use the following command options. For the Ingest Python library, use the following parameters for the `FiltererConfig` object.

* Use `--file-glob` (CLI) or `file_glob` (Python) to specify the list of file extensions to process.
* Use `--max-file-size` (CLI) or `max_file_size` (Python) to specify the maximum size of files to process, in bytes.

## To run this example

The following example processes only `.pdf` and `.eml` files that have a file size of 100 KB or less. To run this example, you should have a directory
with a mixture of files, including at least one `.pdf` file and one `.eml` file, and with at least one of these files having a file size of 100 KB or less.

## Code

<CodeGroup>
  ```bash CLI Ingest v2 theme={null}
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --file-glob "*.pdf,*.eml" \
      --max-file-size 100000 \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.filter import FiltererConfig

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          filterer_config=FiltererConfig(
              file_glob=["*.pdf","*.eml"],
              max_file_size=100000
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "unique_element_ids": True,
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>


# Get chunked elements
Source: https://docs.unstructured.io/open-source/how-to/get-chunked-elements



## Task

You want to get, and print or save, the contents of elements that have been chunked.

## Approach

Chunked elements are typically represented in this format:

```json theme={null}
{
  "type": "CompositeElement",
  "element_id": "3800f7ff047e97cbf0a5f7df7ad52c80",
  "text": "4,5 Exception noted in \u201cSpecial Considerations for Promotional Labeling and Advertising Material\u201d\n\n3",
  "metadata": {
    "filetype": "application/pdf",
    "languages": ["eng"],
    "page_number": 6,
    "orig_elements": "eJy ... Full content omitted for brevity ... x8=",
    "filename": "Portable-Document-Format-Specifications.pdf"
  }
}
```

```json theme={null}
{
  "type": "Table",
  "element_id": "758127b42c51b93b59abf08640d1ccab",
  "text": "Gels and karyotypes High pressure liquid chromatography 600 dpi (8 bit grayscale depth) 300 dpi",
  "metadata": {
    "text_as_html": "<table><tbody><tr><td>Gels and karyotypes</td><td>600 dpi (8 bit grayscale depth)</td></tr><tr><td>High pressure liquid chromatography</td><td>300</td></tr></tbody></table>",
    "filetype": "application/pdf",
    "languages": ["eng"],
    "page_number": 8,
    "orig_elements": "eJy ... Full content omitted for brevity ... MnD",
    "filename": "Portable-Document-Format-Specifications.pdf"
}
```

To get the element's chunked content, extract the contents of the element's `text` field.

The chunked content might not contain all of its associated content. To get all of the elements that were used to derive this chunked content, extract the contents of the element's `orig_elements` field, which is nested inside of its parent `metadata` field.

The contents of the `orig_elements` field is in compressed Base64 gzipped format. To get the content, Base64-decode the bytes, decompress them, and then decode them using UTF-8.

## To run this example

You will need to chunk a document during processing. This example uses a PDF file chunked into 200- to 300-character elements.

## Code

For the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest), you can use the standard Python
[json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
file that the Ingest Python library outputs after the processing is complete.

```python Python theme={null}
import json, base64, zlib
from typing import List, Dict, Any

# Extract the contents of an orig_elements field.
def extract_orig_elements(orig_elements):
    decoded_orig_elements = base64.b64decode(orig_elements)
    decompressed_orig_elements = zlib.decompress(decoded_orig_elements)
    return decompressed_orig_elements.decode('utf-8')

def get_chunked_elements(input_json_file_path: str) -> List[Dict[str, Any]]:
    # Create a dictionary that will hold only
    # a transposed version of the returned elements. 
    # For instance, we just want to capture each element's ID,
    # the chunk's text, and the chunk's associated elements in context.
    orig_elements_dict: List[Dict[str, Any]] = []

    with open(input_json_file_path, 'r') as file:
        file_elements = json.load(file)

    for element in file_elements:
        # For each chunk that has an "orig_elements" field...
        if "orig_elements" in element["metadata"]:
            # ...get the chunk's associated elements in context...
            orig_elements = extract_orig_elements(element["metadata"]["orig_elements"])
            # ...and then transpose it and other associated fields into a separate dictionary.
            orig_elements_dict.append({
                "element_id": element["element_id"],
                "text": element["text"],
                "orig_elements": json.loads(orig_elements)
            })
    
    return orig_elements_dict

if __name__ == "__main__":
    # Source file: https://www.fda.gov/files/drugs/published/Portable-Document-Format-Specifications.pdf
    input_filepath = "local-ingest-output-json/Portable-Document-Format-Specifications.json"
    output_filepath = "local-ingest-output-orig/Portable-Document-Format-Specifications-Orig-Elements-Only.json"

    orig_elements_dict = get_chunked_elements(input_json_file_path = input_filepath)

    # Convert the elements into a JSON object.
    orig_elements_json = json.dumps(orig_elements_dict, indent=2)

    # Write the JSON to a file.
    with open(output_filepath, "w") as file:
        file.write(orig_elements_json)
```

## See also

* [Recovering chunk elements](/open-source/core-functionality/chunking#recovering-chunk-elements)


# Get element contents
Source: https://docs.unstructured.io/open-source/how-to/get-elements



## Task

You want to get, manipulate, and print or save, the contents of the document elements and metadata from the processed data that Unstructured returns.

## Approach

Each element in the document elements contains fields for that element's type, its ID, the extracted text, and associated metadata.

The programmatic approach you take to get these document elements will depend on which tool, SDK, or library you use:

<AccordionGroup>
  <Accordion title="Ingest CLI">
    For the [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli), you can use a tool such as [jq](https://jqlang.github.io/jq/)
    to work with a JSON file that the CLI outputs after the processing is complete.

    For example, the following script uses `jq` to access and print each element's ID, text, and originating file name:

    ```bash Shell theme={null}
    #!/usr/bin/env bash

    JSON_FILE="local-ingest-output/my-file.json"

    jq -r '.[] | "ID: \(.element_id)\nText: \(.text)\nFilename: \(.metadata.filename)\n"' \
        "$JSON_FILE"
    ```
  </Accordion>

  <Accordion title="Ingest Python library">
    For the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest), you can use the standard Python
    [json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
    file that the Ingest Python library outputs after the processing is complete.

    For example, the following code example uses standard Python to access and print each element's ID, text, and originating file name:

    ```python Python theme={null}
    import json

    def parse_json_file(input_file_path: str):
        with open(input_file_path, 'r') as file:
            file_elements = json.load(file)

        for element in file_elements:
            print(f"ID: {element["element_id"]}")
            print(f"Text: {element["text"]}")
            print(f"Filename: {element["metadata"]["filename"]}\n")

    if __name__ == "__main__":
        parse_json_file(
            input_file_path="local-ingest-output/my-file.json"
        )
    ```
  </Accordion>

  <Accordion title="Open-source library">
    For the [Unstructured open-source library](/open-source/introduction/overview), calling the `partition_via_api` function returns a list of elements (`list[Element]`). For example:

    ```python Python theme={null}
    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    # ...
    ```

    You can use standard Python [list operations](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists) on this list.

    You can also use standard Python [looping techniques](https://docs.python.org/3/tutorial/datastructures.html#looping-techniques) on this list to access each element in this list.

    Each individual element has the following attributes:

    * `.text` provides the element's `text` field value as a `str`. See [Element example](/open-source/concepts/document-elements#element-example).
    * `.metadata` provides the element's `metadata` field as an `ElementMetadata` object. See [Metadata](/open-source/concepts/document-elements#metadata).
    * `.category` provides the element's `type` field value as a `str`. See [Element type](/open-source/concepts/document-elements#element-type).
    * `.id` provides the element's `element_id` value as a `str`. See [Element ID](/open-source/concepts/document-elements#element-id).

    In addition, the following methods are available:

    * `.convert_coordinates_to_new_system()` converts the element's location coordinates, if any, to a new coordinate system. See [Element's coordinates](/open-source/concepts/document-elements#elements-coordinates).
    * `.to_dict()` gets the element's content as a standard Python key-value dictionary (`dict[str, Any]`).

    For example:

    ```python Python theme={null}
    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    for element in elements:
        # Do something with each element, for example:
        save_element_to_database(f"{element.id}")
        save_element_to_database(f"{element.text}")
        save_element_to_database(f"{element.metadata.filename}")

    ```

    To serialize this list as a Python dictionary, you can use the `elements_to_dicts` method, for example:

    ```python Python theme={null}
    from unstructured.staging.base import elements_to_dicts

    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    elements_dicts = elements_to_dicts(elements)
    ```

    To serialize this list as JSON, you can use the `elements_to_json` function to convert the list of elements (`Iterable[Element]`) into a JSON-formatted string and then print or save that string. For example:

    ```python Python theme={null}
    from unstructured.staging.base import elements_to_json

    # ...

    elements = partition_via_api(
        filename=input_filepath,
        api_key=os.getenv("UNSTRUCTURED_API_KEY"),
        api_url=os.getenv("UNSTRUCTURED_API_URL"),
        strategy="hi_res"
    )

    json_elements = elements_to_json(
        elements=elements,
        indent=2
    )

    elements_to_json(
        elements=elements,
        indent=2,
        filename=output_filepath
    )
    ```
  </Accordion>
</AccordionGroup>


# Set the OCR agent
Source: https://docs.unstructured.io/open-source/how-to/set-ocr-agent



## Task

You want to specify the type of agent that you use when performing optical character recognition (OCR) on files, such as images and PDFs.

<Note>This task works only with the Unstructured open source library.</Note>

## Approach

Set the environment variable named `OCR_AGENT` to one of the following supported values:

* `unstructured.partition.utils.ocr_models.tesseract_ocr.OCRAgentTesseract` to use Tesseract OCR. This is the default if not otherwise specified.
* `unstructured.partition.utils.ocr_models.paddle_ocr.OCRAgentPaddle` to use Paddle OCR.
* `unstructured.partition.utils.ocr_models.google_vision_ocr.OCRAgentGoogleVision` to use Google Cloud Vision OCR.

Also, be sure to install the corresponding OCR agent and its dependencies, if you have not already done so:

* For Tesseract OCR, [see the dependency list](https://github.com/Unstructured-IO/unstructured/blob/main/requirements/extra-pdf-image.in).
* For Paddle OCR, [see the dependency list](https://github.com/Unstructured-IO/unstructured/blob/main/requirements/extra-paddleocr.in).
* For Google Cloud Vision OCR, [see the dependency list](https://github.com/Unstructured-IO/unstructured/blob/main/requirements/extra-pdf-image.in).

## Example code

This example uses a PNG file with an embedded combination of English and Korean text. This example uses Tesseract OCR.

Language codes will differ depending on the OCR agent you use:

* For Tesseract OCR, [see the language codes list](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/common/lang.py).
* For Paddle OCR, [see the language codes list](https://github.com/Unstructured-IO/unstructured/blob/main/unstructured/partition/lang.py) and [language names list](https://github.com/PaddlePaddle/PaddleOCR/blob/main/doc/doc_en/multi_languages_en.md#language_abbreviations).
* For Google Cloud Vision OCR, [see the language codes list](https://cloud.google.com/vision/docs/languages).

```python Python theme={null}
import json

from unstructured.partition.image import partition_image

# Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/img/english-and-korean.png
# Path to the local file to process, relative to this .py file.
filename = "local-ingest-png/english-and-korean.png"

elements = partition_image(
  filename=filename,
  strategy="ocr_only",
  languages=["eng", "kor"] # Language codes differ by the OCR agent used.
)

# Convert the list of returned elements into a list of dictionaries for printing or saving.
element_dicts = [element.to_dict() for element in elements]

# Print the list.
print(json.dumps(element_dicts, indent=2))

# Or, save the list locally:
#
# file = "local-ingest-output/english-and-korean.json"
#
# with open(file, "w") as file:
#     json.dump(element_dicts, file, indent=2)
```


# Speed up processing of large files and batches
Source: https://docs.unstructured.io/open-source/how-to/speed-up-large-files-batches



When you use Unstructured, here are some techniques that you can try to help speed up the processing of large files and large batches of files.

* Choose your partitioning strategy wisely. For example, if you have simple PDFs that don't have images and tables, you might be able to use the `fast` strategy. Try the `fast` strategy on a few of your documents before you try using the `hi_res` strategy. [Learn more](/open-source/ingestion/ingest-configuration/partition-configuration).
* For processing large numbers of documents, use [ingestion](/open-source/ingestion/overview) and [add CPUs](#adding-cpus).

## Adding CPUs

For speeding up file processing during [ingestion](/open-source/ingestion/overview), the Unstructured CLI and Unstructured Python Ingest enable you to instruct Unstructured to use additional local CPUs where applicable.

Using additional local CPUs applies only to pipeline steps that Unstructured logs as being processed across CPUs. It does not apply to pipeline steps that are logged as being processed asynchronously. To get a list of which operations are processed where, look for the following log messages when you run an ingest pipeline:

* Steps that are processed across CPUs correspond to log messages that read: `processing content across processes`. These steps might benefit by setting a higher number of local CPUs to be used.
* Steps that are processed asynchronously correspond to log messages that read: `processing content across processes`. Any settings to use a higher number of local CPUs are ignored for these steps.

For the Unstructured CLI, you can set `--num-processes` to the maximum number of available local CPUs that you want to use where applicable, for example:

```bash theme={null}
unstructured-ingest \
  local \
    --num-processes <number>
    # ...
```

To get the maximum number of available local logical CPUs that can be used where applicable, see your operating system's documentation.

For Unstructured Python Ingest, you can set the `ProcessorConfig` object's `num_processes` parameter to the maximum number of available local CPUs that you want to use where applicable, for example:

<CodeGroup>
  ```python Python Ingest theme={null}
  from unstructured_ingest.interfaces import ProcessorConfig

  # ...

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(
              num_processes=<number>,
              # ...  
          ),
          # ...
      ).run()
  ```
</CodeGroup>

In Python, to specify the maximum number of available local logical CPUs that can be used where applicable, you can call functions such as [os.cpu\_count](https://docs.python.org/3/library/os.html#os.cpu_count) and [multiprocessing.cpu\_count](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count).

## PDF files

To speed up PDF file processing, the [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) and the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest) provide the following parameters to help speed up processing a large PDF file:

* `split_pdf_page`, when set to true, splits the PDF file on the client side before sending it as batches to Unstructured for processing. The number of pages in each batch is determined internally. Batches can contain between 2 and 20 pages.
* `split_pdf_concurrency_level` is an integer that specifies the number of parallel requests. The default is 5. The maximum is 15. This behavior is ignored unless `split_pdf_page` is also set to true.
* `split_pdf_allow_failed`, when set to true, allows partitioning to continue even if some pages fail.
* `split_pdf_page_range` is a list of two integers that specify the beginning and ending page numbers of the PDF file to be sent. A `ValueError` is raised if the specified range is not valid. This behavior is ignored unless `split_pdf_page` is also set to true.


# Extract tables as HTML
Source: https://docs.unstructured.io/open-source/how-to/text-as-html



## Task

You want to get, save, or show the contents of elements that are represented as HTML, such as tables that are embedded in a PDF document.

## Approach

Extract the contents of an element's `text_as_html` JSON object, which is nested inside of its parent `metadata` object.

## To run this example

You will need a document that is one of the document types that can output the `text_as_html` JSON object. For the list of applicable document types, see the entries in the table at the beginning of [Partitioning](/open-source/core-functionality/partitioning) where "Table Support" is "Yes."

This example uses a PDF file with an embedded table.

## Code

For the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest), you can use the standard Python
[json.load](https://docs.python.org/3/library/json.html#json.load) function to load into a Python dictionary the contents of a JSON
file that the Ingest Python library outputs after the processing is complete.

```python Python theme={null}
import json, os, webbrowser

def get_tables_as_html(
        input_json_file_path: str,
        output_html_dir_path: str
    ):
    with open(input_json_file_path, 'r') as file:
        file_elements = json.load(file)

    # Provide some minimal CSS for better table readability.
    table_css = "<head><style>table, th, td { border: 1px solid; }</style></head>"

    for element in file_elements:
        if "text_as_html" in element["metadata"]:
            # Surround the element's HTML with basic <html> and <body> tags, and add the minimal CSS.
            html_string = f"<!DOCTYPE html><html>{table_css}<body>{element["metadata"]["text_as_html"]}</body></html>"

            # Save the element's HTML to a local file.
            save_path = f"{output_html_dir_path}/{element["element_id"]}.html"
            file = open(save_path, 'w')
            file.write(html_string)
            file.close()

            # View the locally saved file in the local default web browser.
            webbrowser.open_new(f"file:///{os.getcwd()}/{save_path}")

if __name__ == "__main__":
    # Source: https://github.com/Unstructured-IO/unstructured-ingest/blob/main/example-docs/pdf/embedded-images-tables.pdf

    # Specify where to get the local file, relative to this .py file, and
    # where to store the retrieved HTML, relative to this .py file.
    get_tables_as_html(
        input_json_file_path="local-ingest-output/embedded-images-tables.json",
        output_html_dir_path="local-ingest-output/html/"
    )
```


# Astra DB
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/astradb



Batch process all your records to store structured outputs in an Astra DB account.

The requirements are as follows.

<iframe title="YouTube video player" />

* An IBM Cloud account or DataStax account.

  * For an IBM Cloud account, [sign up](https://cloud.ibm.com/registration) for an IBMid, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your IBMid.
  * For a DataStax account, [sign up](https://astra.datastax.com/signup) for a DataStax account, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your DataStax account.

* An Astra DB database in the DataStax account. To create a database:

  a. After you sign in to DataStax, click **Create database**.<br />
  b. Click the **Serverless (vector)** tile, if it is not already selected.<br />
  c. For **Database name**, enter some unique name for the database.<br />
  d. Select a **Provider** and a **Region**, and then click **Create database**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).

* An application token for the database. To create an application token:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Overview** tab, under **Database Details**, in the **Application Tokens** tile, click **Generate Token**.<br />
  c. Enter some **Token description** and select and **Expiration** time period, and then click **Generate token**.<br />
  d. Save the application token that is displayed to a secure location, and then click **Close**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).

* A keyspace in the database. To create a keyspace:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select **Create keyspace**.<br />
  c. Enter some **Keyspace name**, and then click **Add keyspace**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-keyspaces.html#keyspaces).

* A collection in the keyspace.

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * An existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. If Unstructured generates embeddings,
      the new collection's name will be `u<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `u<short-workflow-id`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * For the source connector only, an existing collection is required.
  * For the destination connector only, an existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. The new collection's name will be `unstructuredautocreated`.

  To create a collection yourself:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select the name of the target keyspace.<br />
  c. In the **Collections** list, select **Create collection**.<br />
  d. Enter some **Collection name**.<br />
  e. Turn on **Vector-enabled collection**, if it is not already turned on.<br />
  f. Choose a mode for **Embedding generation method**. See [Astra DB generated embeddings](#astra-db-generated-embeddings).<br />
  g. If you chose **Bring my own**, enter the number of dimensions for the embedding model that you plan to use.<br />
  h. For **Similarity metric**, select **Cosine**.<br />
  i. Click **Create collection**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

The Astra DB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[astradb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `ASTRA_DB_API_ENDPOINT` - The API endpoint for the Astra DB database, represented by `--api-endpoint` (CLI) or `api_endpoint` (Python). To get the endpoint, see the **Database Details > API Endpoint** value on your database's **Overview** tab.
* `ASTRA_DB_APPLICATION_TOKEN` - The database application token value for the database, represented by `--token` (CLI) or `token` (Python). To get the token, see the **Database Details > Application Tokens** box on your database's **Overview** tab.
* `ASTRA_DB_KEYSPACE` - The name of the keyspace for the database, represented by `--keyspace` (CLI) or `keyspace` (Python).
* `ASTRA_DB_COLLECTION` - The name of the collection for the keyspace, represented by `--collection-name` (CLI) or `collection_name` (Python). If no value is provided, see the beginning of this article for the behavior at run time.

Additional settings include:

* For the source connector only, `--fields` (CLI) or `fields` (Python): Optionally, a comma-separated list (CLI) or an array of strings (Python) of fields
  to include in the output. The default is to include all fields, if not otherwise specified.
* For the destination connector only, `--flatten-metadata` (CLI) or `flatten_metadata=True` (Python): Optionally, whether to "flatten" the metadata. Specifically, the metadata key values are
  brought to the top level of the element, and the `metadata` key itself is removed. To not flatten the metadata (the default), specify `--no-flatten_metadata` (CLI) or
  `flatten_metadata=False` (Python). The default is is to not flatten the metadata if not otherwise specified.
* For the destination connector only, `--binary-encode-vectors` (CLI) or `binary_encode_vectors=True` (Python): Optionally, control when to upload vectors in a binary format. By default, vectors are binary encoded (`true`), which is more efficient. To upload vectors as a human-readable list of floats instead, specify `--no-binary-encode-vectors` (CLI) or `binary_encode_vectors=False` (Python). **WARNING**: Disabling this option may make the upload slower!
* For the destination connector only, `--enable-lexical-search` (CLI) or `enable_lexical_search=True` (Python): Optionally, enable lexical and hybrid search.
* For the destination connector only, `--astra-generated-embeddings` (CLI) or `astra_generated_embeddings=True` (Python): Optionally, use an Astra vectorize integration for embeddings.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    astradb \
      --api-endpoint $ASTRA_DB_API_ENDPOINT \
      --token $ASTRA_DB_APPLICATION_TOKEN \
      --collection-name $ASTRA_DB_COLLECTION \
      --keyspace $ASTRA_DB_KEYSPACE \
      --flatten-metadata \
      --no-binary-encode-vectors \
      --no-enable-lexical-search \
      --no-astra-generated-embeddings
      
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.astradb import (
      AstraDBConnectionConfig,
      AstraDBAccessConfig,
      AstraDBUploadStagerConfig,
      AstraDBUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(
          ),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AstraDBConnectionConfig(
              access_config=AstraDBAccessConfig(
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN")
              )
          ),
          stager_config=AstraDBUploadStagerConfig(
              flatten_metadata=True,
              enable_lexical_search=False,
              astra_generated_embeddings=False
          ),
          uploader_config=AstraDBUploaderConfig(
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              batch_size=20,
              record_id_key="record_id",
              binary_encode_vectors=False
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Azure
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/azure



Batch process all your records to store structured outputs in an Azure Storage account.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Azure Storage account requirements:

<iframe title="YouTube video player" />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  * **Read** and **List** for reading from the container only.
  * **Write** and **List** for writing to the container only.
  * **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these requirements:

* An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

  <iframe title="YouTube video player" />

* An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

  <iframe title="YouTube video player" />

  <iframe title="YouTube video player" />

* The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

  Create an SAS token (recommended):

  <iframe title="YouTube video player" />

  Get an access key or connection string:

  <iframe title="YouTube video player" />

The Azure Storage account connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[azure]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `AZURE_STORAGE_REMOTE_URL` - The Azure Storage remote URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

  The remote URL takes the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* `AZURE_STORAGE_ACCOUNT_NAME` - The name of the Azure Storage account, represented by `--account-name` (CLI) or `account_name` (Python).

* `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_STORAGE_CONNECTION_STRING`, or `AZURE_STORAGE_SAS_TOKEN` - The name of the key, connection string, or SAS token for the Azure Storage account, depending on your security configuration, represented by `--account-key` (CLI) or `account_key` (Python), `--connection-string` (CLI) or `connection_string` (Python), and `--sas_token` (CLI) or `sas_token` (Python), respectively.

Now call the Unstructured Ingest CLI or Unstructured Ingest Python.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    azure \
      --remote-url $AZURE_STORAGE_REMOTE_URL \
      --account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --account-key $AZURE_STORAGE_ACCOUNT_KEY
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.azure import (
      AzureConnectionConfig,
      AzureAccessConfig,
      AzureUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          uploader_config=AzureUploaderConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Azure AI Search
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/azure-ai-search



Batch process all your records to store structured outputs in an Azure AI Search account.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Azure AI Search requirements:

<iframe title="YouTube video player" />

Here are some more details about these requirements:

* The endpoint and API key for Azure AI Search. [Create an endpoint and API key](https://learn.microsoft.com/azure/search/search-create-service-portal).
* The name of the index in Azure AI Search. [Create an index](https://learn.microsoft.com/rest/api/searchservice/create-index).

  <iframe title="YouTube video player" />

  The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents
  that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  You can adapt the following index schema example for your own needs. Be sure to replace `<number-of-dimensions>`
  (in three locations in the following example) with the number of dimensions of the embedding model you are using:

  ```json theme={null}
  {
    "name": "elements-index",
    "fields": [
      {
        "name": "id",
        "type": "Edm.String",
        "key": true
      },
      {
        "name": "record_id",
        "type": "Edm.String",
        "filterable": true
      },
      {
        "name": "element_id",
        "type": "Edm.String"
      },
      {
        "name": "text",
        "type": "Edm.String",
        "searchable": true
      },
      {
        "name": "type",
        "type": "Edm.String"
      },
      {
        "name": "metadata",
        "type": "Edm.ComplexType",
        "fields": [
          {
            "name": "orig_elements",
            "type": "Edm.String"
          },
          {
            "name": "category_depth",
            "type": "Edm.Int32"
          },
          {
            "name": "parent_id",
            "type": "Edm.String"
          },
          {
            "name": "attached_to_filename",
            "type": "Edm.String"
          },
          {
            "name": "filetype",
            "type": "Edm.String"
          },
          {
            "name": "last_modified",
            "type": "Edm.DateTimeOffset"
          },
          {
            "name": "is_continuation",
            "type": "Edm.Boolean"
          },
          {
            "name": "file_directory",
            "type": "Edm.String"
          },
          {
            "name": "filename",
            "type": "Edm.String"
          },
          {
            "name": "data_source",
            "type": "Edm.ComplexType",
            "fields": [
              {
                "name": "url",
                "type": "Edm.String"
              },
              {
                "name": "version",
                "type": "Edm.String"
              },
              {
                "name": "date_created",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "date_modified",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "date_processed",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "permissions_data",
                "type": "Edm.String"
              },
              {
                "name": "record_locator",
                "type": "Edm.String"
              }
            ]
          },
          {
            "name": "coordinates",
            "type": "Edm.ComplexType",
            "fields": [
              {
                "name": "system",
                "type": "Edm.String"
              },
              {
                "name": "layout_width",
                "type": "Edm.Double"
              },
              {
                "name": "layout_height",
                "type": "Edm.Double"
              },
              {
                "name": "points",
                "type": "Edm.String"
              }
            ]
          },
          {
            "name": "languages",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "page_number",
            "type": "Edm.String"
          },
          {
            "name": "links",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "page_name",
            "type": "Edm.String"
          },
          {
            "name": "link_urls",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "link_texts",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "sent_from",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "sent_to",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "subject",
            "type": "Edm.String"
          },
          {
            "name": "section",
            "type": "Edm.String"
          },
          {
            "name": "header_footer_type",
            "type": "Edm.String"
          },
          {
            "name": "emphasized_text_contents",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "emphasized_text_tags",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "text_as_html",
            "type": "Edm.String"
          },
          {
            "name": "regex_metadata",
            "type": "Edm.String"
          },
          {
            "name": "detection_class_prob",
            "type": "Edm.Double"
          }
        ]
      },
      {
        "name": "embeddings",
        "type": "Collection(Edm.Single)",
        "dimensions": <number-of-dimensions>,
        "vectorSearchProfile": "embeddings-config-profile"
      }
    ],
    "vectorSearch": {
      "algorithms": [
        {
          "name": "hnsw-<number-of-dimensions>",
          "kind": "hnsw",
          "hnswParameters": {
            "m": 4,
            "efConstruction": 400,
            "efSearch": 500,
            "metric": "cosine"
          }
        }
      ],
      "profiles": [
        {
          "name": "embeddings-config-profile",
          "algorithm": "hnsw-<number-of-dimensions>"
        }
      ]
    },
    "semantic": {
      "configurations": [
        {
          "name": "default-semantic-config",
          "prioritizedFields": {
            "titleField": null,
            "prioritizedContentFields": [
              { "fieldName": "text" }
            ],
            "prioritizedKeywordsFields": []
          }
        }
      ]
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [Search indexes in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-an-index)
  * [Schema of a search index](https://learn.microsoft.com/azure/search/search-what-is-an-index#schema-of-a-search-index)
  * [Example index schema](https://learn.microsoft.com/rest/api/searchservice/create-index#examples)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

The Azure AI Search connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[azure-ai-search]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `AZURE_SEARCH_ENDPOINT` - The endpoint URL for Azure AI Search, represented by `--endpoint` (CLI) or `endpoint` (Python).
* `AZURE_SEARCH_API_KEY` - The API key for Azure AI Search, represented by `--key` (CLI) or `key` (Python).
* `AZURE_SEARCH_INDEX` - The name of the index for Azure AI Search, represented by `--index` (CLI) or `index` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    azure-ai-search \
      --key $AZURE_SEARCH_API_KEY \
      --endpoint $AZURE_SEARCH_ENDPOINT \
      --index $AZURE_SEARCH_INDEX
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.azure_ai_search import (
      AzureAISearchAccessConfig,
      AzureAISearchConnectionConfig,
      AzureAISearchUploadStagerConfig,
      AzureAISearchUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureAISearchConnectionConfig(
              access_config=AzureAISearchAccessConfig(
                  key=os.getenv("AZURE_SEARCH_API_KEY")
              ),
              endpoint=os.getenv("AZURE_SEARCH_ENDPOINT"),
              index=os.getenv("AZURE_SEARCH_INDEX")
          ),
          stager_config=AzureAISearchUploadStagerConfig(),
          uploader_config=AzureAISearchUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Box
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/box



Batch process all your records to store structured outputs in a Box account.

The requirements are as follows.

<iframe title="YouTube video player" />

1. Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2. A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

   * In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
   * In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
   * Click **Save Changes** before continuing.

4. Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5. Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

   * In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
   * **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
   * Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6. The private key configuration JSON file for the Box Custom App, or a string that contains this file's contents.

   * To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location.
   * To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as
     a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself).
     To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
     In this command, replace `<path-to-downloaded-key-file>` with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.

     * For macOS or Linux:

       ```text theme={null}
       tr -d '\n' < <path-to-downloaded-key-file>
       ```

     * For Windows:

       ```text theme={null}
       (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
       ```

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

Permissions are defined in Box through *collaborations*, which function similar to access control lists. Collaboration objects are attached to folders, and those collaborations cascade down to everything inside that folder. A file's effective permissions come from the chain of ancestor folders above it. For more information, see [Collaborations overview](https://developer.box.com/guides/collaborations/) in the Box developer documentation.

To determine a file's effective permissions, Unstructured traverses the full folder hierarchy for each file, collecting collaborations at every level, and merging them. It also includes collaborations applied directly to the file itself. For more information, see [List folder collaborations](https://developer.box.com/reference/get-folders-id-collaborations/)  and [List file collaborations](https://developer.box.com/reference/get-files-id-collaborations/) in the Box developer documentation.

When Unstructured compiles a file's effective permissions:

* If the same user or group appears at multiple levels with different roles, Unstructured chooses the least-restrictive role.
* Unstructured does not include the following in the permission metadata. No sentinel value is written into the permission metadata.
  * `all_users_group`: Box's built-in group representing every user in the enterprise.
  * `is_access_only`: Collaboration property that determines whether, for collaborators, to display the items in the All Files list and let them see the path to the root folder for the shared item.
* Unstructured only includes collaborations with a status of `accepted`. It does not include collaborations with a status of `pending` or `rejected`.

Unstructured includes a maximum of 500 permissions per file.

Unstructured maps Box roles to `read`, `update`, and `delete` access:

| Box role             | Box access                                               | Unstructured permission metadata |
| -------------------- | -------------------------------------------------------- | -------------------------------- |
| `owner`              | Full access                                              | `read`, `update`, `delete`       |
| `co-owner`           | Full access except transfer ownership                    | `read`, `update`, `delete`       |
| `editor`             | View, download, upload, edit, delete, copy, move, rename | `read`, `update`                 |
| `viewer`             | Preview, download, comment                               | `read`                           |
| `previewer`          | Preview                                                  | `read`                           |
| `viewer uploader`    | Preview, download, upload                                | `read`                           |
| `previewer uploader` | Preview, upload                                          | `read`                           |
| `uploader`           | Upload and see item names                                | excluded entirely                |

Note the following:

* Unstructured excludes the `uploader` role because it confers no `read`, `update`, or `delete` access.
* Unstructured maps the Box `editor` role to `read` and `update` access but not `delete`, even though that role does allow deleting files.

For more information, see [Understanding collaborator permission level](https://support.box.com/hc/en-us/articles/360044196413-Understanding-Collaborator-Permission-Levels) at Box Support.

### Identifier format

Users and groups are identified by the numeric ID that Box assigns them. To retrieve information about a specific user or group, use appropriate Box API with the corresponding ID:

* [Get user](https://developer.box.com/reference/get-users-id/)
* [Get group](https://developer.box.com/reference/get-groups-id/)

### Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["11446688", "11336699"],
                            "groups": ["11223344"]
                        }
                    },
                    {
                        "update": {
                            "users": ["11446688"],
                            "groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

The Box connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[box]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `BOX_APP_CONFIG` - The local path to the downloaded private key configuration JSON file for the Box Custom App,
  or a single-line string that contains the contents of this file, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
* `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).
  This URL must take the format `box://<path/to/folder/in/account>`.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    box \
      --box-app-config $BOX_APP_CONFIG \
      --remote-url $BOX_REMOTE_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.box import (
      BoxAccessConfig,
      BoxConnectionConfig,
      BoxIndexerConfig,
      BoxUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=BoxConnectionConfig(
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG")
              )
          ),
          uploader_config=BoxUploaderConfig(
              remote_url=os.getenv("BOX_REMOTE_URL")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Chroma
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/chroma



Batch process all your records to store structured outputs in a Chroma account.

The requirements are as follows.

* A Chroma server. You can connect to a [hosted Chroma Cloud server](https://trychroma.com/signup) or [deploy your own server](https://docs.trychroma.com/deployment).
* The Chroma server's hostname or IP address, and the server's port number. For Chroma Cloud, the host is `api.trychroma.com`, and the port number is `8000`.
* If you are using local Chroma, the [path to the folder where you store your database](https://docs.trychroma.com/docs/run-chroma/persistent-client).
* The name of the tenant that you want to access on the server.
* The name of the database that you want to access in the tenant.
* The name of the collection that you want to access in the database.

The Chroma connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[chroma]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `CHROMA_HOST` - The , represented by `--host` (CLI) or `host` (Python). Chroma Cloud uses `api.trychroma.com`.
* `CHROMA_PORT` - The , represented by `--port` (CLI) or `port` (Python). Chroma Cloud uses port `8000`.
* `CHROMA_TENANT` - The name of the tenant that you want to access on the Chroma server, represented by `--tenant` (CLI) or `tenant` (Python).
* `CHROMA_DATABASE` - The name of the database that you want to access in the tenant, represented by `--database` (CLI) or `database` (Python).
* `CHROMA_COLLECTION` - The name of the collection that you want to access in the database, represented by `--collection-name` (CLI) or `collection_name` (Python).

Additional settings include:

* `--path` (CLI) or `path` (Python): The location where Chroma is persisted if you are not connecting through HTTP.
* `--settings` (CLI) or `settings` (Python): A dictionary of settings to communicate with the Chroma server, for example: `'{"persist_directory":"./chroma-persist"}'`.
* `--headers` (CLI) or `headers` (Python): A dictionary of headers to send to the Chroma server, for example: `'{"Authorization":"Basic()"}'`.
* `--ssl` (CLI) or `ssl` (Python): True to use SSL for the connection.

In order to use Chroma Cloud, you should have this configuration:

* `host=api.trychroma.com`
* `port=8000`
* Get your `tenant` ID from the [Chroma dashboard](https://trychroma.com).
* `headers={'x-chroma-token': 'YOUR_API_KEY'}`
* `ssl` should be enabled.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --work-dir $WORK_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    chroma \
      --host $CHROMA_HOST \
      --port $CHROMA_PORT \
      --collection-name $CHROMA_COLLECTION \
      --tenant $CHROMA_TENANT \
      --database $CHROMA_DATABASE \
      --settings "{\"persist_directory\":\"./chroma-persist\"}" \
      --headers "{\"Authorization\":\"Basic()\"}" \
      --batch-size 80 \
      --ssl
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.chroma import (
      ChromaAccessConfig,
      ChromaConnectionConfig,
      ChromaUploadStagerConfig,
      ChromaUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=ChromaConnectionConfig(
              access_config=ChromaAccessConfig(
                  settings={"persist_directory":"./chroma-persist"},
                  headers={"Authorization":"Basic()"}
              ),
              host=os.getenv("CHROMA_HOST"),
              port=os.getenv("CHROMA_PORT"),
              tenant=os.getenv("CHROMA_TENANT"),
              database=os.getenv("CHROMA_DATABASE"),
              collection_name=os.getenv("CHROMA_COLLECTION")
          ),
          stager_config=ChromaUploadStagerConfig(),
          uploader_config=ChromaUploaderConfig()
      ).run()

  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Couchbase
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/couchbase



[Couchbase](https://couchbase.com) is a Distributed NoSQL Cloud Database. Couchbase embraces AI with coding assistance for developers, and vector search for their applications.

Batch process all your records to store structured outputs in a Couchbase database.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Couchbase Capella clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe title="YouTube video player" />

For Couchbase Capella, you will need:

* A [Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier).
* A [Couchbase Capella cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started).
* A [bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket),
  [scope](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-scope),
  and [collection](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-collection)
  on the cluster.
* The cluster's [public connection string](https://docs.couchbase.com/cloud/get-started/connect.html#connect-from-sdk-cbsh-cli-or-ide).
* The [cluster access name (username) and secret (password)](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials).
* [Incoming IP address allowance](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) for the cluster.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

For a local Couchbase server, you will need:

* [Installation of a local Couchbase server](https://docs.couchbase.com/server/current/getting-started/start-here.html).
* [Connection details](https://docs.couchbase.com/server/current/guides/connect.html) to the local Couchbase server.

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

The Couchbase DB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[couchbase]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables are required for the Couchbase Connector:

* `CB_CONN_STR` - The Connection String for the Couchbase server, represented by `--connection-string` (CLI) or `connection_string` (Python).
* `CB_USERNAME` - The username for the Couchbase server, represented by `--username` (CLI) or `username` (Python).
* `CB_PASSWORD` - The password for the Couchbase server, represented by `--password` (CLI) or `password` (Python).
* `CB_BUCKET` - The name of the bucket in the Couchbase server, represented by `--bucket` (CLI) or `bucket` (Python).
* `CB_SCOPE` - The name of the scope in the bucket, represented by `--scope` (CLI) or `scope` (Python).
* `CB_COLLECTION` - The name of the collection in the scope, represented by `--collection` (CLI) or `collection` (Python).

Additional available settings include:

* `--collection-id` (CLI) or `collection_id` in `CouchbaseDownloaderConfig` (Python) - Optional for the source connector. The\
  unique key of the ID field in the collection. The default is `id` if not otherwise specified.
  [Learn more](https://docs.couchbase.com/server/current/learn/services-and-indexes/indexes/indexing-and-query-perf.html#introduction-document-keys).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    couchbase \
      --connection-string $CB_CONN_STR \
      --username $CB_USERNAME \
      --password $CB_PASSWORD \
      --bucket $CB_BUCKET \
      --scope $CB_SCOPE \
      --collection $CB_COLLECTION \
      --num-processes 2 \
      --batch-size 80
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.couchbase import (
      CouchbaseAccessConfig,
      CouchbaseConnectionConfig,
      CouchbaseUploadStagerConfig,
      CouchbaseUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=CouchbaseConnectionConfig(
              access_config=CouchbaseAccessConfig(
                  password=os.getenv("CB_PASSWORD"),
              ),
              connection_string=os.getenv("CB_CONN_STR"),
              username=os.getenv("CB_USERNAME"),
              bucket=os.getenv("CB_BUCKET"),
              scope=os.getenv("CB_SCOPE"),
              collection=os.getenv("CB_COLLECTION")
          ),
          stager_config=CouchbaseUploadStagerConfig(),
          uploader_config=CouchbaseUploaderConfig(batch_size=100)
      ).run()
  ```
</CodeGroup>

To understand how [vector search](https://www.couchbase.com/products/vector-search/) works in Couchbase, refer to this [tutorial](https://developer.couchbase.com/tutorial-python-langchain-pdf-chat) and the [Couchbase docs](https://docs.couchbase.com/cloud/vector-search/vector-search.html)

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Delta Tables in Databricks
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/databricks-delta-table



<Note>
  This article covers connecting Unstructured to Delta Tables in Databricks.

  For information about connecting Unstructured to Delta Tables in Amazon S3 instead, see
  [Delta Tables in Amazon S3](/open-source/ingestion/destination-connectors/delta-table).

  For information about connecting Unstructured to Databricks Volumes instead, see
  [Databricks Volumes](/open-source/ingestion/destination-connectors/databricks-volumes).
</Note>

Batch process all your records to store structured outputs in a Delta Table in Databricks.

The Delta Tables in Databricks connector provides two implementations:

* The *SQL-based* implementation enables you to have Unstructured write its processed data directly to a table in Unity Catalog.
  If you use this implementation, you do not need to provide a volume as described in the following requirements.
* The *volume-based* implementation enables you to have Unstructured write its processed data as a file to a volume in Unity Catalog. Unstructured then
  uses that file in the volume to write the file's data to a table in Unity Catalog. The file remains in the volume.

The requirements are as follows.

* [Account and resources](#accounts-and-resources)
* [Unity Catalog resources](#unity-catalog-resources)
* [Authentication and permissions](#authentication-and-permissions)

### Accounts and resources

* A Databricks account on [Amazon Web Services (AWS)](https://docs.databricks.com/getting-started/free-trial.html), [Microsoft Azure (Azure)](https://learn.microsoft.com/azure/databricks/getting-started/), or [Google Cloud Platform (GCP)](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html), [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* One of the following compute resources within the workspace:

  * A SQL warehouse for [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html), [Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create), or [GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html).

    The following video shows how to create a SQL warehouse if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the warehouse's owner to use it:

    <iframe title="YouTube video player" />

  * An all-purpose cluster for [AWS](https://docs.databricks.com/compute/use-compute.html), [Azure](https://learn.microsoft.com/azure/databricks/compute/use-compute), or [GCP](https://docs.gcp.databricks.com/compute/use-compute.html).

    The following video shows how to create an all-purpose cluster if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the cluster's owner to use it:

    <iframe title="YouTube video player" />

* The SQL warehouse's or cluster's **Server Hostname** and **HTTP Path** values for [AWS](https://docs.databricks.com/integrations/compute-details.html),
  [Azure](https://learn.microsoft.com/azure/databricks/integrations/compute-details), or
  [GCP](https://docs.gcp.databricks.com/integrations/compute-details.html).

* Unity Catalog enabled in the workspace for [AWS](https://docs.databricks.com/data-governance/unity-catalog/get-started.html),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/get-started), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/get-started.html).

### Unity Catalog resources

Within Unity Catalog:

* A catalog for [AWS](https://docs.databricks.com/catalogs/create-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html).
* A schema for [AWS](https://docs.databricks.com/schemas/create-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) within that catalog.

  <Note>
    A *schema* was formerly known as a *database* in Unity Catalog.
  </Note>
* A table for [AWS](https://docs.databricks.com/tables/managed.html), [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or [GCP](https://docs.gcp.databricks.com/tables/managed.html) within that schema.

  How you create and specify the table depends on whether you want document metadata stored as nested JSON in a single column, or have Unstructured flatten the metadata and store each field as a separate columns. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.

  * If you choose to store document metadata as a JSON blob, you can specify a table, or have the connector attempt to create a table for you automatically at run time.

    ***To specify an existing table***

    In the connector settings as described later in this article, specify the name of an existing table.

    The following video shows how to create a catalog, schema, and a table in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

    <iframe title="YouTube video player" />

    If you want to use an existing table or create one yourself beforehand, this table must contain at minimum the following column names and their data types:

    ```text theme={null}
    CREATE TABLE IF NOT EXISTS <catalog_name>.<schema_name>.<table_name> (
        id STRING NOT NULL PRIMARY KEY,
        record_id STRING NOT NULL,
        element_id STRING NOT NULL,
        text STRING,
        embeddings ARRAY<FLOAT>,
        type STRING,
        metadata VARIANT
    );
    ```

    <Info>
      In Databricks, a table's *schema* is different than a *schema* in a catalog-schema object relationship in Unity Catalog.
    </Info>

    <Info>
      The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
    </Info>

    ***To have the connector create a table***

    You can have the connector attempt to create a table for you automatically at run time. In the connector settings as described later in this article, do one of the following:

    * Specify the name of the table that you want the connector to attempt to create within the specified catalog and schema.
    * Leave the table name blank. The connector will attempt to create a table within the specified catalog and schema.

      For [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview), the table is named `u<short-workflow-id>`.
      For the [Unstructured Ingest CLI and Ingest Python library](/open-source/ingestion/overview), the table is named `unstructuredautocreated`.

    The connector will attempt to create the table on behalf of the related Databricks workspace user or Databricks managed service principal that is referenced in the connector settings, as described later in these requirements.
    If successful, the table's owner is set as the related Databricks workspace user or Databricks managed service principal. The owner will have all Unity Catalog
    privileges on the table by default. No other Databricks workspace users or Databricks managed service principals will have any privileges on the table by default.

    <Warning>
      If the table's parent schema is not owned by the same Databricks workspace user or Databricks managed service principal that is
      referenced in the connector settings, then you should grant the new table's owner the `CREATE TABLE` privilege on that parent schema
      before the connector attempts to create the table. Otherwise, table creation could fail.
    </Warning>

    <Note>
      Using dashes (`-`) in the names of catalogs, schemas, and tables might cause isolated issues with the connector. It is
      recommended to use underscores (`_`) instead of dashes in the names of catalogs, schemas, and tables.
    </Note>

  * If you choose to flatten the document metadata, you must create the table before you configure your connector. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.

### Authentication and permissions

* For Databricks managed service principal authentication (using Databricks OAuth M2M) to the workspace:

  * A Databricks managed service principal.
    This service principal must have the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse.
  * The service principal's **UUID** (or **Client ID** or **Application ID**) value.
  * The OAuth **Secret** value for the service principal.

  To get this information, see Steps 1-3 of the instructions for [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
  [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

  <Note>
    For Azure Databricks, this connector only supports Databricks managed service principals for authentication.
    Microsoft Entra ID managed service principals are not supported.
  </Note>

  The following video shows how to create a Databricks managed service principal if you do not already have one available:

  <iframe title="YouTube video player" />

* For Databricks personal access token authentication to the workspace, the
  Databricks personal access token value for
  [AWS](https://docs.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat#azure-databricks-personal-access-tokens-for-workspace-users), or
  [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users).
  This token must be for the workspace user who
  has the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse,

  The following video shows how to create a Databricks personal access token if you do not already have one available:

  <iframe title="YouTube video player" />

* The Databricks workspace user or Databricks managed service principal must have the following *minimum* set of permissions and privileges to write to an
  existing volume or table in Unity Catalog. If the owner of these is that Databricks workspace user or Databricks managed service principal, then
  they will have all necessary permissions and privileges by default. If the owner is someone else, then the following permissions and privileges must be
  explicitly granted to them before using the connector:

  * To use an all-purpose cluster for access, `Can Restart` permission on that cluster. Learn how to check and set cluster permissions for
    [AWS](https://docs.databricks.com/compute/clusters-manage.html#compute-permissions),
    [Azure](https://learn.microsoft.com/azure/databricks/compute/clusters-manage#cluster-level-permissions), or
    [GCP](https://docs.gcp.databricks.com/compute/clusters-manage.html#compute-permissions).

  * To use a SQL warehouse for access, `Can use` permission on that SQL warehouse. Learn how to check and set SQL warehouse permissions for
    [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html#manage-a-sql-warehouse),
    [Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create#manage), or
    [GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html#manage-a-sql-warehouse).

  * To access a Unity Catalog volume, the following privileges:

    * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
    * `USE SCHEMA` on the volume's parent schema in Unity Catalog.
    * `READ VOLUME` and `WRITE VOLUME` on the volume.

    Learn how to check and set Unity Catalog privileges for
    [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
    [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
    [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

  * To access a Unity Catalog table, the following privileges:

    * `USE CATALOG` on the table's parent catalog in Unity Catalog.
    * `USE SCHEMA` on the table's parent schema in Unity Catalog.
    * To create a new table, `CREATE TABLE` on the table's parent schema in Unity Catalog.
    * If the table already exists, `MODIFY` and `SELECT` on the table.

    Learn how to check and set Unity Catalog privileges for
    [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
    [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
    [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

* Within Unity Catalog, a volume
  for [AWS](https://docs.databricks.com/volumes/utility-commands.html),
  [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands),
  or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html). The volume can be in the same
  schema as the table, or the volume and table can be in separate schemas. In either case, both of these
  schemas must share the same parent catalog.

  <Note>
    Using dashes (`-`) in the names of volumes might cause isolated issues with the connector. It is
    recommended to use underscores (`_`) instead of dashes in the names of volumes.
  </Note>

  The following video shows how to create a catalog, schema, and a volume in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

  <iframe title="YouTube video player" />

  ## Storing document metadata

  Unstructured offers the following options for storing [document metadata](/concepts/document-elements#metadata) in the destination table:

  * Store the metadata as a single nested JSON field.
  * *Flatten* the metadata by writing each metadata field as its own typed, queryable column.

  Each of these approaches has its advantages. In general, storing the metadata as a JSON blob works for most use cases, including when:

  * Performing dot.notation queries on the stored JSON is sufficient for your needs.
  * Document metadata schemas vary across file sources. When flattening document metadata, Unstructured drops fields that do not match existing columns in the schema.
  * You want the connector to automatically generate the destination table. This option is not supported when flattening document metadata.

  Flattening the metadata and storing it in multiple columns is most useful when:

  * You want to query individual metadata fields directly using standard SQL, without JSON parsing.
  * The business intelligence or analytics tools you are using require columnar data.

  Storing metadata as a JSON blob is the default.

  To store metadata as a JSON blob, when configuring the connector uncheck **Flatten Metadata** (in the Unstructured Pipelines), or set `flatten_metadata` to `false` (in the Unstructured API).  To flatten the metadata, check **Flatten Metadata**, or set `flatten_metadata` to `true`.

  ### How Unstructured flattens metadata

  If you choose to have Unstructured flatten document metadata, you must create the Delta Table to use as the destination, and specify it when creating the connector. In order to prevent possible data loss, Unstructured will not automatically create a new table.

  Considerations to keep in mind when creating the Delta Table:

  * The table must contain a column for each metadata field you want to store. Any metadata field that does not have a corresponding column in the table is silently dropped, although the event is written to the logs. For more information, see [Logging and monitoring](/business/security-compliance/overview#logging-and-monitoring).
  * Do not declare metadata columns as `NOT NULL`. Missing metadata values are written as `NULL`.
  * Unstructured passes values through as their JSON-native type: strings, numbers, boolean, and so on. For example, no special formatting is applied to timestamp values.
  * Metadata fields that are lists are not further flattened. Lists remain single columns.

  ### Metadata flattening example

  The following example demonstrates how Unstructured flattens metadata into separate columns. Consider the following metadata:

  ```json theme={null}
  {
    "metadata": {
      "category_depth": 1,
      "data_source": {
        "url": "s3://my-source-bucket/path/chart-and-table.pdf",
        "version": "864ae40b0719e976e98ba0a7b9fcba92",
        "record_locator": {
          "protocol": "s3",
          "remote_file_path": "s3://my-source-bucket/path/"
        }
      },
      "languages": ["eng"]
    }
  }
  ```

  When flattening metadata, Unstructured generates a field name comprised of the full path to that field within the metadata structure, from the outermost object to the field itself. For example, `protocol`, which is included in the `record_locator` object, which is in turn within `data_source`, becomes `data_source_record_locator_protocol`:

  ```json theme={null}
  {
    "category_depth": 1,
    "data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
    "data_source_version": "864ae40b0719e976e98ba0a7b9fcba92",
    "data_source_record_locator_protocol": "s3",
    "data_source_record_locator_remote_file_path": "s3://my-source-bucket/path/",
    "languages": ["eng"]
  }
  ```

The Delta Tables in Databricks connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[databricks-delta-tables]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DATABRICKS_HOST` - The Databricks cluster's or SQL warehouse's **Server Hostname** value, represented by `--server-hostname` (CLI) or `server_hostname` (Python).

* `DATABRICKS_HTTP_PATH` - The cluster's or SQL warehouse's **HTTP Path** value, represented by `--http-path` (CLI) or `http_path` (Python).

* `DATABRICKS_TOKEN` - For Databricks personal access token authentication, the token's value, represented by `--token` (CLI) or `token` (Python).

* `DATABRICKS_CLIENT_ID` - For Databricks managed service principal authenticaton, the service principal's **UUID** (or **Client ID** or **Application ID**) value, represented by `--client-id` (CLI) or `client_id` (Python).

* `DATABRICKS_CLIENT_SECRET` - For Databricks managed service principal authenticaton, the service principal's OAuth **Secret** value, represented by `--client-secret` (CLI) or `client_secret` (Python).

* `DATABRICKS_CATALOG` - The name of the catalog in Unity Catalog, represented by `--catalog` (CLI) or `catalog` (Python).

* `DATABRICKS_DATABASE` - The name of the schema (formerly known as a database) inside of the catalog for the target table, represented by `--database` (CLI) or `database` (Python). The default is `default` if not otherwise specified.

  If you are also using a volume, and the target table and volume are in the same schema (formerly known as a database), then `DATABRICKS_DATABASE` and `DATABRICKS_SCHEMA` will have the same values.

* `DATABRICKS_TABLE` - The name of the table inside of the schema (formerly known as a database), represented by `--table-name` (CLI) or `table_name` (Python).

  * If a table name is specified, but a table with that name does not exist within the specified schema (formerly known as a database), the connector attempts to create a table with that name within that schema.
  * If no table name is specified, the connector attempts to create a table named `u<short-workflow-id>` within the specified schema (formerly known as a database).

  See the beginning of this article for additional technical requirements before having the connector attempt to create a table.

<Note>
  Using dashes (`-`) in the names of catalogs, schemas (formerly known as databases), tables, and volumes might cause isolated issues with the connector. It is
  recommended to use underscores (`_`) instead of dashes in the names of catalogs, schemas, tables, and volumes.
</Note>

For the SQL-based implementation, add these environment variables:

* `DATABRICKS_RECORD_ID_KEY` - The name of the column that uniquely identifies each record in the table, represented by `--record-id-key` (CLI) or `record_id_key` (Python).
* The maximum number of records to be transmitted per batch, represented by `--batch-size` (CLI) or `batch_size` (Python). The default is `50` if not otherwise specified.

For the volume-based implementation, add these environment variables:

* `DATABRICKS_SCHEMA` - The name of the schema (formerly known as a database) inside of the catalog for the target volume, represented by `--schema` (CLI) or `schema` (Python). The default is `default` if not otherwise specified.

  If the target volume and table are in the same schema (formerly known as a database), then `DATABRICKS_SCHEMA` and `DATABRICKS_SCHEMA` will have the same values.

* `DATABRICKS_VOLUME` - The name of the volume inside of the schema (formerly known as a database), represented by `--volume` (CLI) or `volume` (Python).

* `DATABRICKS_VOLUME_PATH` - Optionally, a specific path inside of the volume that you want to start accessing from, starting from the volume's root, represented by `--volume-path` (CLI) or `volume_path` (Python). The default is to start accessing from the volume's root if not otherwise specified.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI (SQL-Based) theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For authenticating with Databricks personal access tokens:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    databricks-delta-tables \
      --server-hostname $DATABRICKS_HOST \
      --http-path $DATABRICKS_HTTP_PATH \
      --token $DATABRICKS_TOKEN \
      --catalog $DATABRICKS_CATALOG \
      --database $DATABRICKS_DATABASE \
      --table-name $DATABRICKS_TABLE
      --record-id-key $DATABRICKS_RECORD_ID_KEY \
      --batch-size 50

  # For authenticating with Databricks managed service principals:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    databricks-delta-tables \
      --server-hostname $DATABRICKS_HOST \
      --http-path $DATABRICKS_HTTP_PATH \
      --client-id $DATABRICKS_CLIENT_ID \
      --client-secret $DATABRICKS_CLIENT_SECRET \
      --catalog $DATABRICKS_CATALOG \
      --database $DATABRICKS_DATABASE \
      --table-name $DATABRICKS_TABLE
      --record-id-key $DATABRICKS_RECORD_ID_KEY \
      --batch-size 50
  ```

  ```bash CLI (Volume-Based) theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For authenticating with Databricks personal access tokens:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    databricks-volume-delta-tables \
      --server-hostname $DATABRICKS_HOST \
      --http-path $DATABRICKS_HTTP_PATH \
      --token $DATABRICKS_TOKEN \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --database $DATABRICKS_DATABASE \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH \
      --table-name $DATABRICKS_TABLE
    
  # For authenticating with Databricks managed service principals:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    databricks-volume-delta-tables \
      --server-hostname $DATABRICKS_HOST \
      --http-path $DATABRICKS_HTTP_PATH \
      --client-id $DATABRICKS_CLIENT_ID \
      --client-secret $DATABRICKS_CLIENT_SECRET \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --database $DATABRICKS_DATABASE \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH \
      --table-name $DATABRICKS_TABLE
  ```

  ```python Python Ingest (SQL-Based) theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.databricks_delta_tables import (
      DatabricksDeltaTablesConnectionConfig,
      DatabricksDeltaTablesAccessConfig,
      DatabricksDeltaTablesUploadStagerConfig,
      DatabricksDeltaTablesUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          
          # For authenticating with Databricks personal access tokens.
          destination_connection_config=DatabricksDeltaTablesConnectionConfig(
              access_config=DatabricksDeltaTablesAccessConfig(
                  token=os.getenv("DATABRICKS_TOKEN"),
              ),
              server_hostname=os.getenv("DATABRICKS_HOST"),
              http_path=os.getenv("DATABRICKS_HTTP_PATH")
          ),

          # For authenticating with Databricks managed service principals.
          # destination_connection_config=DatabricksDeltaTablesConnectionConfig(
          #     access_config=DatabricksDeltaTablesAccessConfig(
          #         client_id=os.getenv("DATABRICKS_CLIENT_ID"),
          #         client_secret=os.getenv("DATABRICKS_CLIENT_SECRET")
          #     ),
          #     server_hostname=os.getenv("DATABRICKS_HOST"),
          #     http_path=os.getenv("DATABRICKS_HTTP_PATH")
          # ),

          stager_config=DatabricksDeltaTablesUploadStagerConfig(),
          uploader_config=DatabricksDeltaTablesUploaderConfig(
              catalog=os.getenv("DATABRICKS_CATALOG"),
              database=os.getenv("DATABRICKS_DATABASE"),
              table_name=os.getenv("DATABRICKS_TABLE"),
              record_id_key=os.getenv("DATABRICKS_RECORD_ID_KEY"),
              batch_size=50    
          )
      ).run()
  ```

  ```python Python Ingest (Volume-Based) theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.databricks_delta_tables import (
      DatabricksDeltaTablesConnectionConfig,
      DatabricksDeltaTablesAccessConfig,
      DatabricksDeltaTablesUploadStagerConfig
  )

  from unstructured_ingest.processes.connectors.databricks.volumes_table import (
      DatabricksVolumeDeltaTableUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          
          # For authenticating with Databricks personal access tokens.
          destination_connection_config=DatabricksDeltaTablesConnectionConfig(
              access_config=DatabricksDeltaTablesAccessConfig(
                  token=os.getenv("DATABRICKS_TOKEN"),
              ),
              server_hostname=os.getenv("DATABRICKS_HOST"),
              http_path=os.getenv("DATABRICKS_HTTP_PATH")
          ),

          # For authenticating with Databricks managed service principals.
          # destination_connection_config=DatabricksDeltaTablesConnectionConfig(
          #     access_config=DatabricksDeltaTablesAccessConfig(
          #         client_id=os.getenv("DATABRICKS_CLIENT_ID"),
          #         client_secret=os.getenv("DATABRICKS_CLIENT_SECRET")
          #     ),
          #     server_hostname=os.getenv("DATABRICKS_HOST"),
          #     http_path=os.getenv("DATABRICKS_HTTP_PATH")
          # ),

          stager_config=DatabricksDeltaTablesUploadStagerConfig(),
          uploader_config=DatabricksVolumeDeltaTableUploaderConfig(
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              database=os.getenv("DATABRICKS_DATABASE"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
              table_name=os.getenv("DATABRICKS_TABLE")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Databricks Volumes
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/databricks-volumes



<Note>
  This article covers connecting Unstructured to Databricks Volumes.

  For information about connecting Unstructured to Delta Tables in Databricks instead, see
  [Delta Tables in Databricks](/open-source/ingestion/destination-connectors/databricks-delta-table).
</Note>

Batch process all your records to store structured outputs in Databricks Volumes.

The requirements are as follows.

* A Databricks account on [AWS](https://docs.databricks.com/getting-started/free-trial.html),
  [Azure](https://learn.microsoft.com/azure/databricks/getting-started/), or
  [GCP](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or
  [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* The workspace's URL. Get the workspace URL for
  [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
  [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
  or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

  Examples:

  * AWS: `https://<workspace-id>.cloud.databricks.com`
  * Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
  * GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

  <Note>
    Do not add a trailing slash (`/`) to the workspace URL.
  </Note>

* The Databricks authentication details. For more information, see the documentation for
  [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
  or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

  For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), the following Databricks authentication types are supported:

  * Databricks OAuth machine-to-machine (M2M) authentication for\
    [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

    You will need the **Client ID** (or **UUID** or **Application** ID) and OAuth **Secret** (client secret) values for the corresponding service principal.
    Note that for Azure, only Databricks managed service principals are supported. Microsoft Entra ID managed service principals are not supported.

    The following video shows how to create a Databricks managed service principal:

    <iframe title="YouTube video player" />

  * Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html).

    You will need the personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  For [Unstructured Ingest](/open-source/ingestion/overview), the following Databricks authentication types are supported:

  * For Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html): The personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  * For username and password (basic) authentication ([AWS](https://docs.databricks.com/archive/dev-tools/basic.html) only): The user's name and password values.

  * For OAuth machine-to-machine (M2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html)): The client ID and OAuth secret values for the corresponding service principal.

  * For OAuth user-to-machine (U2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-u2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-u2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-u2m.html)): No additional values.

  * For Azure managed identities (formerly Managed Service Identities (MSI) authentication) ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-mi) only): The client ID value for the corresponding managed identity.

  * For Microsoft Entra ID service principal authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-sp) only): The tenant ID, client ID, and client secret values for the corresponding service principal.

  * For Azure CLI authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-cli) only): No additional values.

  * For Microsoft Entra ID user authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/user-aad-token) only): The Entra ID token for the corresponding Entra ID user.

  * For Google Cloud Platform credentials authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-creds.html) only): The local path to the corresponding Google Cloud service account's credentials file.

  * For Google Cloud Platform ID authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-id.html) only): The Google Cloud service account's email address.

* The name of the parent catalog in Unity Catalog for
  [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
  [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
  [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html) for the volume.

* The name of the parent schema (formerly known as a database) in Unity Catalog for
  [AWS](https://docs.databricks.com/schemas/create-schema.html),
  [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
  [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) for the volume.

* The name of the volume in Unity Catalog for [AWS](https://docs.databricks.com/tables/managed.html),
  [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
  [GCP](https://docs.gcp.databricks.com/tables/managed.html), and optionally any path in that volume that you want to access directly, beginning with the volume's root.

* The Databricks workspace user or service principal must have the following *minimum* set of privileges to read from or write to the
  existing volume in Unity Catalog:

  * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
  * `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
  * `READ VOLUME` and `WRITE VOLUME` on the volume.

  The following videos show how to create and set privileges for a catalog, schema (formerly known as a database), and volume in Unity Catalog.

  <iframe title="YouTube video player" />

  Learn more about how to check and set Unity Catalog privileges for
  [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

The Databricks Volumes connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[databricks-volumes]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DATABRICKS_HOST` - The Databricks host URL, represented by `--host` (CLI) or `host` (Python).

  <Note>
    Do not add a trailing slash (`/`) to the host URL.
  </Note>

* `DATABRICKS_CATALOG` - The Databricks catalog name for the Volume, represented by `--catalog` (CLI) or `catalog` (Python).

* `DATABRICKS_SCHEMA` - The Databricks schema name for the Volume, represented by `--schema` (CLI) or `schema` (Python). If not specified, `default` is used.

* `DATABRICKS_VOLUME` - The Databricks Volume name, represented by `--volume` (CLI) or `volume` (Python).

* `DATABRICKS_VOLUME_PATH` - Any optional path to access within the volume, specified by `--volume-path` (CLI) or `volume_path` (Python).

Environment variables based on your authentication type, depending on your cloud provider:

* For Databricks personal access token authentication (AWS, Azure, and GCP):

  * `DATABRICKS_TOKEN` - The personal access token, represented by `--token` (CLI) or `token` (Python).

* For username and password (basic) authentication (AWS only): The user's name and password values.

  * `DATABRICKS_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
  * `DATABRICKS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

* For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.

  * `DATABRICKS_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--client-id` (CLI) or `client_id` (Python).
  * `DATABRICKS_CLIENT_SECRET` - The client ID and OAuth secret values for the corresponding service principal, represented by `--client-secret` (CLI) or `client_secret` (Python).

* For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional environment variables.

* For Azure managed identities (MSI) authentication (Azure only):

  * `ARM_CLIENT_ID` - The client ID value for the corresponding managed identity, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
  * If the target identity has not already been added to the workspace, then you must also specify the
    `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

* For Microsoft Entra ID service principal authentication (Azure only):

  * `ARM_TENANT_ID` - The tenant ID value for the corresponding service principal, represented by `--azure-tenant-id` (CLI) or `azure_tenant_id` (Python).
  * `ARM_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
  * `ARM_CLIENT_SECRET` - The client secret value for the corresponding service principal, represented by `--azure-client-secret` (CLI) or `azure_client_secret` (Python).
  * If the service principal has not already been added to the workspace, then you must also specify the
    `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

* For Azure CLI authentication (Azure only): No additional environment variables.

* For Microsoft Entra ID user authentication (Azure only):

  * `DATABRICKS_TOKEN` - The Entra ID token for the corresponding Entra ID user, represented by `--token` (CLI) or `token` (Python).

* For Google Cloud Platform credentials authentication (GCP only):

  * `GOOGLE_CREDENTIALS` - The local path to the corresponding Google Cloud service account's credentials file, represented by `--google-credentials` (CLI) or `google_credentials`

* For Google Cloud Platform ID authentication (GCP only):

  * `GOOGLE_SERVICE_ACCOUNT` - The Google Cloud service account's email address, represented by `--google-service-account` (CLI) or `google_service_account` (Python).

* Alternatively, you can store the preceding settings in a local
  [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth/config-profiles.html) and then just
  refer to the profile's name:

  * `DATABRICKS_PROFILE` - The name of the Databricks configuration profile, represented by `--profile` (CLI) or `profile` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    databricks-volumes \
      --profile $DATABRICKS_PROFILE \
      --host $DATABRICKS_HOST \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  # For all supported Databricks authentication types, you can import this:
  from unstructured_ingest.processes.connectors.databricks.volumes import (
      DatabricksVolumesConnectionConfig,
      DatabricksVolumesAccessConfig,
      DatabricksVolumesUploaderConfig
  )

  # Alternatively, for supported Databricks on AWS authentication types only, you can import this:
  # from unstructured_ingest.processes.connectors.databricks.volumes_aws import (
  #     DatabricksAWSVolumesConnectionConfig,
  #     DatabricksAWSVolumesAccessConfig,
  #     DatabricksAWSVolumesUploaderConfig
  # )

  # Alternatively, for supported Azure Databricks authentication types only, you can import this:
  # from unstructured_ingest.processes.connectors.databricks.volumes_azure import (
  #     DatabricksAzureVolumesConnectionConfig,
  #     DatabricksAzureVolumesAccessConfig,
  #     DatabricksAzureVolumesUploaderConfig
  # )

  # Alternatively, for supported Databricks on Google Cloud authentication types only, you can import this:
  # from unstructured_ingest.processes.connectors.databricks.volumes_gcp import (
  #     DatabricksGoogleVolumesConnectionConfig,
  #     DatabricksGoogleVolumesAccessConfig,
  #     DatabricksGoogleVolumesUploaderConfig
  # )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          # For specifying a Databricks configuration profile:
          destination_connection_config=DatabricksVolumesConnectionConfig(
              access_config=DatabricksVolumesAccessConfig(profile=os.getenv("DATABRICKS_PROFILE")),
              host=os.getenv("DATABRICKS_HOST"),
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
          uploader_config=DatabricksVolumesUploaderConfig(overwrite=True)
          # Other examples:
          #
          # For Databricks on AWS, with Databricks personal access token authentication:
          # destination_connection_config=DatabricksAWSVolumesConnectionConfig(
          #     access_config=DatabricksAWSVolumesAccessConfig(token=os.getenv("DATABRICKS_TOKEN")),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAWSVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
          #
          # For Azure Databricks, with Microsoft Entra ID service principal authentication:
          # destination_connection_config=DatabricksAzureVolumesConnectionConfig(
          #     access_config=DatabricksAzureVolumesAccessConfig(
          #         azure_client_id=os.getenv("ARM_CLIENT_ID"),
          #         azure_client_secret=os.getenv("ARM_CLIENT_SECRET"),
          #         azure_tenant_id=os.getenv("ARM_TENANT_ID")   
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAzureVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
          #
          # For Databricks on Google Cloud, with Google Cloud Platform credentials authentication:
          # destination_connection_config=DatabricksGoogleVolumesConnectionConfig(
          #     access_config=DatabricksGoogleVolumesAccessConfig(
          #         google_service_account=os.getenv("GOOGLE_CREDENTIALS")
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          # uploader_config=DatabricksAWSVolumesUploaderConfig(
          #     catalog=os.getenv("DATABRICKS_CATALOG"),
          #     schema=os.getenv("DATABRICKS_SCHEMA"),
          #     volume=os.getenv("DATABRICKS_VOLUME"),
          #     volume_path=os.getenv("DATABRICKS_VOLUME_PATH"),
          #     overwrite=True
          # )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Delta Tables in Amazon S3
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/delta-table



<Note>
  This article covers connecting Unstructured to Delta Tables in Amazon S3. For information about
  connecting Unstructured to Delta Tables in Databricks instead, see
  [Delta Tables in Databricks](/open-source/ingestion/destination-connectors/databricks-delta-table).
</Note>

Batch process all your records to store structured outputs in a Delta Table in an Amazon S3 bucket.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Amazon S3 requirements to store Delta Tables:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account.

For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* For authenticated bucket write access or both, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. To grant permissions,
  attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
  To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Provide an AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, provide the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Delta table output format

A Delta table consists of Parquet files that contain data and a transaction log that stores metadata about the transactions.
[Learn more](https://delta-io.github.io/delta-rs/how-delta-lake-works/architecture-of-delta-table/).

The Delta Tables in Amazon S3 destination connector generates the following output within the specified path to the S3 bucket (or the specified folder within the bucket):

* Initially, one Parquet (`.parquet`) file per file in the source location. For example, for a file in the source location named `my-file.pdf`, an associated
  file with the extension `.parquet` is generated. Various kinds of file transactions can result in additional Parquet files being generated. These Parquet filenames are automatically generated by the Delta Lake engine and are not meant to be manually modified.
* A folder named `_delta_log` that contains metadata and change history about the `.parquet` files. As Parquet files are added to, changed, or removed from
  the specified bucket or folder path, the `_delta_log` folder is updated with any related metadata and change history details.

Together, this set of Parquet files and their associated `_delta_log` folder (and its contents) describe a single, versioned Delta table. Because of this, Unstructured recommends the following usage best practices:

* In the source location, each set of source files that is to be considered as a unit for change management purposes should be controlled by a unique, dedicated
  Delta Tables in S3 destination connector. This connector should reference a unique, dedicated output folder within the bucket. Having
  multiple workflows refer to different sets of source files, yet all share the same Delta table, could results in data loss or table corruption.
* Avoid directly modifying, adding, or deleting Parquet data files or the `_delta_log` folder within a Delta table's directory. This can lead to data loss or table corruption.
* If you need to copy or move a Delta table to a different location,
  you must move or copy its entire set of Parquet files and its associated `_delta_log` folder (and its contents) together as a unit.
  Note that the copied or moved Delta table will
  no longer be controlled by the original Delta Tables in S3 destination connector.

The Delta Table connector dependencies for Amazon S3:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[delta-table]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`, represented by `--table-uri` (CLI) or `table_uri` (Python).
* `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--aws-access-key-id` (CLI) or `aws_access_key` (Python).
* `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--aws-secret-access-key` (CLI) or `aws_secret_access_key` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    delta-table \
      --aws-access-key-id $AWS_ACCESS_KEY_ID \
      --aws-secret-access-key $AWS_SECRET_ACCESS_KEY \
      --table-uri $AWS_S3_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.delta_table import (
      DeltaTableConnectionConfig,
      DeltaTableAccessConfig,
      DeltaTableUploadStagerConfig,
      DeltaTableUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DeltaTableConnectionConfig(
              access_config=DeltaTableAccessConfig(
                  aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
                  aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              table_uri=os.getenv("AWS_S3_URL")
          ),
          stager_config=DeltaTableUploadStagerConfig(),
          uploader_config=DeltaTableUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Dropbox
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/dropbox



Batch process all your records to store structured outputs in a Dropbox account.

The requirements are as follows.

<iframe title="YouTube video player" />

1. A [Dropbox account](https://www.dropbox.com/try/teams).

2. A Dropbox app for your Dropbox account. To create a Dropbox app, do the following:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click **Create app**.<br />
   d) For **Choose an API**, select **Scoped access**.<br />
   e) For **Choose the type of access you need**, select **App folder**.<br />
   f) Enter a name for your app, and then click **Create app**.<br />
   g) On the app's **Permissions** tab, under **Files and folders**, check the boxes labelled **files.content.read** or **files.content.write** or both,
   depending on whether you want to read files, write files, or both. Then click **Submit**.<br />
   h) On the app's **Settings** tab, note the value of the **App folder name** field. This is the name of the app folder that Dropbox will create under the `Apps` top-level folder in your Dropbox
   account that the Dropbox app will use for access. If you change the value of **App folder name** field here, Dropbox will create an app folder with that name under the `Apps` top-level folder instead.<br />
   i) Under **OAuth 2**, next to **Generated access token**, click **Generate**. Copy the value of this access token. You should only click **Generate** after you have completed all of the preceding steps first.
   This is because the access token is scoped to the specific app folder and settings at the time the access token is generated. If you change the app folder name or any of the permissions later,
   you should regenerate the access token.<br />

   <Warning>
     Access tokens are valid for **only four hours** after they are created. After this four-hour period, you can no longer use the expired access token.
     Dropbox does not allow the creation of access tokens that are valid for more than four hours.

     To replace an expired access token, you must first generate a *refresh token* for the corresponding access token. To learn how to generate an access token and its corresponding refresh token,
     see [Replace an expired access token](#replace-an-expired-access-token), later in this article.

     If you do not already have the corresponding refresh token for an existing access token, or if you lose a refresh token after you generate it,
     you must generate a new access token and its corresponding refresh token.

     Instead of continualy replacing expired access tokens yourself, you can have Unstructured do it for you as needed; just supply Unstructured
     with the refresh token along with the Dropbox app's **App key** and **App secret** values.
     To learn how to supply these to Unstructured, look for mentions of "refresh token," "app key," and "app secret" in the connector settings later in this article.
   </Warning>

3. The app folder that your Dropbox app will use for access can be found in your Dropbox account under the `Apps` top-level folder. For example, if the value of the **App folder name**
   field above is `my-folder`, then the app folder that your Dropbox app will use for access can be found under `https://dropbox.com/home/Apps/my-folder`

   <img alt="The my-folder app folder under the Apps top-level folder" />

   <Warning>
     Your Dropbox app will *not* have access to upload or download files from the root of the app folder. Instead, you *must* create a subfolder inside of the app folder for your Dropbox
     app to upload or download files from. You will use the name of that subfolder when specifying your remote URL in the next step. For example, if your Dropbox app uses an app folder named `my-folder`
     for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then the subfolder that your Dropbox app will upload and download files from
     can be found under `https://dropbox.com/home/Apps/my-folder/data`

     <img alt="The data subfolder under the my-folder subfolder" />
   </Warning>

4. Note the remote URL to your subfolder inside of the app folder, which takes the format `dropbox://<subfolder-name>`. For example,
   if your Dropbox app uses an app folder named `my-folder` for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then
   the remote URL is `dropbox://data`

   <img alt="The data subfolder under the my-folder subfolder" />

## Replace an expired access token

Dropbox app access tokens are valid for **only four hours**. After this time, you can no longer use the expired access token.

To have Unstructured automatically replace expired access tokens on your behalf, do the following:

<iframe title="YouTube video player" />

1. Get the app key and app secret values for your Dropbox app. To do this:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click your Dropbox app's icon.<br />
   d) On the **Settings** tab, next to **App key**, copy the value of the app key.<br />
   e) Next to **App secret**, click **Show**, and then copy the value of the app secret.

2. Use your web browser to browse to the following URL, replacing `<app-key>` with the app key for your Dropbox app:

   ```text theme={null}
   https://www.dropbox.com/oauth2/authorize?client_id=<app-key>&response_type=code&token_access_type=offline
   ```

3. Click **Continue**.

4. Click **Allow**.

5. In the **Access code generated** tile, copy the access code that is shown.

6. Use the [curl](https://curl.se/) utility in your Terminal or Command Prompt, or use a REST API client such as
   [Postman](https://www.postman.com/product/api-client/), to make the following REST API call, replacing the following placeholders:

   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.
   * Replace `<access-code>` with the access code that you just copied.

   ```text theme={null}
   curl --location --request POST 'https://api.dropbox.com/oauth2/token' \
   --user '<app-key>:<app-secret>' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data-urlencode 'code=<access-code>' \
   --data-urlencode 'grant_type=authorization_code'
   ```

7. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token.
   * The value of `refresh_token` is the refresh token that can be used to replace this access token much faster and easier next time.
     If you lose this refresh token, you must go back to Step 2.

   For the [Unstructured Pipelines](/pipelines/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the connector settings **Refresh token** field.
   * Add the `<app-key>` value to the connector settings **App key** field.
   * Add the `<app-secret>` value to the connector settings **App secret** field.

   For the [Unstructured API](/api-reference/overview) and [Unstructured Ingest](/open-source/ingestion/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the `refresh_token` parameter.
   * Add the `<app-key>` value to the `app_key` parameter.
   * Add the `<app-secret>` value to the connector settings `app_secret` parameter.

8. If for some reason you need to manually replace the expired access token yourself instead of having Unstructured do it for you, you can use the refresh token that you just copied to get a new access token:

   * Replace `<refresh-token>` with the refresh token.
   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.

   ```text theme={null}
   curl https://api.dropbox.com/oauth2/token \
   --data refresh_token=<refresh-token> \
   --data grant_type=refresh_token \
   --data client_id=<app-key> \
   --data client_secret=<app-secret>
   ```

9. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token. In the connector, replace the old,
     expired access token value with this new, valid access token value.
   * The value of `refresh_token` is the new, valid refresh token. To replace the expired access token yourself, go back to Step 8.

The Dropbox connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[dropbox]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DROPBOX_REMOTE_URL` - The remote URL to the target subfolder inside of the app folder for the Dropbox app, represented by `--remote-url` (CLI) or `remote_url` (Python).
* `DROPBOX_ACCESS_TOKEN` - The value of the access token for the Dropbox app that is associated with the target app folder, represented by `--token` (CLI) or `token` (Python).
  Provide this only if for some reason you do not want Unstructured to automatically refresh expired access tokens.

To have Unstructured automatically refresh expired Dropbox App access tokens on your behalf, do not provide an access token. Instead, provide the following environment variables:

* `DROPBOX_REFRESH_TOKEN` - The value of the refresh token for the corresponding access token, represented by `--refresh-token` (CLI) or `refresh_token` (Python).
* `DROPBOX_APP_KEY` - The app key for the Dropbox app, represented by `--app-key` (CLI) or `app_key` (Python).
* `DROPBOX_APP_SECRET` - The app secret for the Dropbox app, represented by `--app-secret` (CLI) or `app_secret` (Python).

Now call the Unstructured Ingest CLI or Unstructured Ingest Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    dropbox \
      --refresh-token $DROPBOX_REFRESH_TOKEN \
      --app-key $DROPBOX_APP_KEY \
      --app-secret $DROPBOX_APP_SECRET \
      --remote-url $DROPBOX_REMOTE_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.dropbox import (
      DropboxAccessConfig,
      DropboxConnectionConfig,
      DropboxUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DropboxConnectionConfig(
              access_config=DropboxAccessConfig(
                  refresh_token=os.getenv("DROPBOX_REFRESH_TOKEN"),
                  app_key=os.getenv("DROPBOX_APP_KEY"),
                  app_secret=os.getenv("DROPBOX_APP_SECRET")
              )
          ),
          uploader_config=DropboxUploaderConfig(
              remote_url=os.getenv("DROPBOX_REMOTE_URL")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# DuckDB
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/duckdb



Batch process all your records to store structured outputs in a DuckDB installation.

The requirements are as follows.

* A [DuckDB installation](https://duckdb.org/docs/installation).

* A [persistent database](https://duckdb.org/docs/connect/overview.html#persistent-database), for example by running the
  [DuckDB CLI](https://duckdb.org/docs/api/cli) command `duckdb <my-database-filename>.db` or
  `duckdb <my-database-filename>.duckdb`, replacing `<my-database-filename>` with the name of the target file.

* The path to the target persistent database file.

* A schema in the target database.

  * [Create a schema](https://duckdb.org/docs/sql/statements/create_schema.html).
  * You can list available schemas and their parent catalogs by running the following DuckDB CLI command:

    ```sql theme={null}
    SELECT * FROM information_schema.schemata;
    ```

  The DuckDB connector uses the default schema name of `main` if not otherwise specified.

* A table in the target schema.

  * [Create a table](https://duckdb.org/docs/sql/statements/create_table).
  * You can list available tables in a schema by running the following DuckDB CLI commands, replacing the target catalog and schema names:

    ```sql theme={null}
    USE <catalog_name>.<schema_name>;
    SHOW TABLES;
    ```

  The DuckDB connector uses the default table name of `elements` if not otherwise specified.

  For maximum compatibility, Unstructured recommends the following table schema:

  ```sql theme={null}
  CREATE TABLE elements (
      id VARCHAR,
      element_id VARCHAR,
      text TEXT,
      embeddings FLOAT[],
      type VARCHAR,
      system VARCHAR,
      layout_width DECIMAL,
      layout_height DECIMAL,
      points TEXT,
      url TEXT,
      version VARCHAR,
      date_created INTEGER,
      date_modified INTEGER,
      date_processed DOUBLE,
      permissions_data TEXT,
      record_locator TEXT,
      category_depth INTEGER,
      parent_id VARCHAR,
      attached_filename VARCHAR,
      filetype VARCHAR,
      last_modified TIMESTAMP,
      file_directory VARCHAR,
      filename VARCHAR,
      languages VARCHAR[],
      page_number VARCHAR,
      links TEXT,
      page_name VARCHAR,
      link_urls VARCHAR[],
      link_texts VARCHAR[],
      sent_from VARCHAR[],
      sent_to VARCHAR[],
      subject VARCHAR,
      section VARCHAR,
      header_footer_type VARCHAR,
      emphasized_text_contents VARCHAR[],
      emphasized_text_tags VARCHAR[],
      text_as_html TEXT,
      regex_metadata TEXT,
      detection_class_prob DECIMAL,
      partitioner_type VARCHAR
  );
  ```

  You can list the schema of a table by running the following DuckDB CLI commands, replacing the target catalog, schema, and table names:

  ```sql theme={null}
  USE <catalog_name>.<schema_name>;
  DESCRIBE TABLE <table_name>;
  ```

The DuckDB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[duckdb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DUCKDB_DATABASE` - The path to the target DuckDB persistent database file with the extension `.db` or `.duckdb`, represented by `--database` (CLI) or `database` (Python).
* `DUCKDB_DB_SCHEMA` - The name of the target schema in the database, represented by `--db-schema` (CLI) or `db_schema` (Python).
* `DUCKDB_TABLE` - The name of the target table in the schema, represented by `--table` (CLI) or `table` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    duckdb \
      --database $DUCKDB_DATABASE \
      --db-schema $DUCKDB_DB_SCHEMA \
      --table $DUCKDB_TABLE
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.duckdb.duckdb import (
      DuckDBAccessConfig,
      DuckDBConnectionConfig,
      DuckDBUploadStagerConfig,
      DuckDBUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=DuckDBConnectionConfig(
              access_config=DuckDBAccessConfig(),
              database=os.getenv("DUCKDB_DATABASE"),
              db_schema=os.getenv("DUCKDB_DB_SCHEMA"),
              table=os.getenv("DUCKDB_TABLE")
          ),
          stager_config=DuckDBUploadStagerConfig(),
          uploader_config=DuckDBUploaderConfig(batch_size=50)
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Elasticsearch
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/elasticsearch



Batch process all your records to store structured outputs in Elasticsearch.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Elastic Cloud instances are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

* For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

  <iframe title="YouTube video player" />

* For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

  <iframe title="YouTube video player" />

* You will need the name of the index on the instance.

  For the destination connector, if you need to create an index, you can use for example the following `curl` command. Replace the following placeholders:

  * Replace `<host>:<port>` with the instance's host identifier and port number.

  * Replace one of the following, and remove the other:

    * For an instance's user and password combination, replace `<username>` with your Elasticsearch or Elastic Cloud instance's user name, replace `<password>` with your password, and then remove `--user "Authorization: ApiKey <your-api-key>"`.
    * For an Elastic Cloud API key, replace `<your-api-key>` with your Elastic Cloud API key, and then remove `--user "<username>:<password>"`.

  * Replace `<index-name>` with the name of the new index on the instance.

  * Replace `<index-schema>` with the schema for the index. A schema is optional; see the explanation
    following this `curl` command for more information.

  ```bash theme={null}
  curl --request PUT "<host>:<port>/<index-name>" \
  --user "<username>:<password>" \
  --user "Authorization: ApiKey <your-api-key>" \
  [--header "Content-Type: application/json" \
  --data '<index-schema>']
  ```

  To learn more, see [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

  For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
  Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
  to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  For objects in the `metadata` field that Unstructured produces and that you want to store in an Elasticsearch or Elastic Cloud index, you must create fields in your index's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    },
    "entities": {
      "items": [
        {
          "entity": "HOME FOR FUTURE",
          "type": "ORGANIZATION"
        },
        {
          "entity": "221 Queen Street, Melbourne VIC 3000",
          "type": "LOCATION"
        }
      ],
      "relationships": [
        {
          "from": "HOME FOR FUTURE",
          "relationship": "based_in",
          "to": "221 Queen Street, Melbourne VIC 3000"
        }
      ]
    }
  }
  ```

  You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
  required by Unstructured whenever you create your own index schema:

  * `element_id`
  * `record_id`, which is required by Unstructured for intelligent record updates.
  * `type`, which is not required, but highly recommended.
  * `text`
  * `embeddings` if embeddings are generated; make sure to set `dims` to the same number of dimensions as the embedding model generates.

  ```json theme={null}
  {
    "mappings": {
      "properties": {
        "element_id": {
          "type": "keyword"
        },
        "record_id": {
          "type": "text"
        },
        "text": {
          "type": "text"
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "embeddings": {
          "type": "dense_vector",
          "dims": 1536,
          "index": true,
          "similarity": "cosine"
        },
        "metadata": {
          "properties": {
            "is_extracted": {
              "type": "boolean"
            },
            "coordinates-points": {
              "type": "float"
            },
            "coordinates-system": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates-layout_width": {
              "type": "long"
            },
            "coordinates-layout_height": {
              "type": "long"
            },
            "filetype": {
              "type": "keyword"
            },
            "languages": {
              "type": "keyword"
            },
            "page_number": {
              "type": "integer"
            },
            "image_mime_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "data_source-url": {
              "type": "keyword"
            },
            "data_source-record_locator-protocol": {
              "type": "keyword"
            },
            "data_source-record_locator-remote_file_path": {
              "type": "keyword"
            },
            "entities-items": {
              "properties": {
                "entity": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "entities-relationships": {
              "properties": {
                "from": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "relationship": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "to": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
  * [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
  * [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  <iframe title="YouTube video player" />

* For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console or [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

  <Note>
    When generating an Elastic Cloud service instance's API key through the Elasticsearch Service web console, be sure after you click **Create API key** to select **Encoded** to get the API key's value.
    (The **Encoded** API key value is the Base64 encoding of the UTF-8 representation of the API key's ID and key value joined by a colon \[:].)
  </Note>

* For self-managed Elasticsearch, you will need:

  * The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).

  * If you're using basic authentication to the self-managed instance, the user's name and password.

  * If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

    <Note>
      When generating an API key for a self-managed Elasticsearch instance by calling [POST /\_security/api\_key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key), be sure to use
      the `encoded` value in the response payload. (This `encoded` value is the Base64 encoding of the UTF-8 representation of the payload's `id` and `api_key` values joined by a colon \[:]). Do not use the `api_key` value by itself.
    </Note>

  * If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

The Elasticsearch connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[elasticsearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `ELASTICSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
* `ELASTICSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using Elastic Cloud to connect to ElasticSearch:

* `ELASTIC_CLOUD_ID` - The instance's Cloud ID, represented by `--cloud-id` (CLI) or `cloud_id` (Python).
* `ELASTIC_CLOUD_API_KEY` - The corresponding Cloud API key value, represented by `--api-key-id` (CLI) or `api_key_id` (Python).

If you're using basic authentication to the instance:

* `ELASTICSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
* `ELASTICSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using token-based authentication to the instance instead:

* `ELASTICSEARCH_BEARER_TOKEN` - The bearer token's value, represented by `--bearer-auth` (CLI) or `bearer_auth` (Python).
* `ELASTIC_CLOUD_API_KEY_ID` - The API key's value, represented by `--api-key` (CLI) or `api_key` (Python).

If you're using certificates:

* `ELASTICSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) file, represented by `--ca-certs` (CLI) or `ca_certs` (Python).
* `ELASTICSEARCH_SSL_ASSERT_FINGERPRINT` - The certificate's fingerprint, represented by `--ssl-assert-fingerprint` or `ssl_assert_fingerprint` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash Shell theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 4 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    elasticsearch \
      --hosts $ELASTICSEARCH_HOST \
      --username $ELASTICSEARCH_USERNAME \
      --password $ELASTICSEARCH_PASSWORD \
      --index-name $ELASTICSEARCH_INDEX_NAME \
      --ca-certs $ELASTICSEARCH_CA_CERTS \
      --ssl-assert-fingerprint $ELASTICSEARCH_SSL_ASSERT_FINGERPRINT \
      --num-processes 2
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.elasticsearch.elasticsearch import (
      ElasticsearchConnectionConfig,
      ElasticsearchAccessConfig,
      ElasticsearchUploadStagerConfig,
      ElasticsearchUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=ElasticsearchConnectionConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS")
          ),
          stager_config=ElasticsearchUploadStagerConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME")),
          uploader_config=ElasticsearchUploaderConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Google Cloud Storage
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/google-cloud-service



Batch process all your records to store structured outputs in Google Cloud Service.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

* A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
  [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:
    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```
  * For Windows:
    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

  This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

  * `Storage Object Viewer` for bucket read access.
  * `Storage Object User` for bucket write access.
  * The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

  To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
  in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

The Google Cloud Storage connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[gcs]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `GCS_SERVICE_ACCOUNT_KEY` - The Google Cloud service account key for Google Cloud Storage, represented by `--service-account-key` (CLI) or `service_account_key` (Python).
* `GCS_REMOTE_URL` - The Google Cloud Storage bucket URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    gcs \
      --service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --remote-url $GCS_REMOTE_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.gcs import (
      GcsConnectionConfig,
      GcsAccessConfig,
      GcsUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=GcsConnectionConfig(
              access_config=GcsAccessConfig(
                  service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
              )
          ),
          uploader_config=GcsUploaderConfig(remote_url=os.getenv("GCS_REMOTE_URL"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# IBM watsonx.data
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/ibm-watsonxdata



<Tip>
  The IBM watsonx.data destination connector relies on an Apache Iceberg-based catalog within the watsonx.data data store instance.
  Apache Iceberg is suitable for managed data storage and cataloging, but not for embedding storage or semantic similarity
  queries. For embedding storage and semantic similarity queries, Unstructured recommends that you use the following destination connectors
  instead:

  * [Astra DB](/open-source/ingestion/destination-connectors/astradb)
  * [Milvus](/open-source/ingestion/destination-connectors/milvus) on IBM watsonx.data
</Tip>

Batch process all your records to store structured outputs in IBM watsonx.data.

The requirements are as follows.

* An [IBM Cloud account](https://cloud.ibm.com/login). [Create an IBM Cloud account](https://cloud.ibm.com/registration) if you do not already have one.

* An API key for the IBM Cloud account. If you do not have one already, create one as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. In the top navigation bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
  3. On the sidebar, under **Manage identities**, click **API keys**.
  4. With the **View** list showing **My IBM Cloud API keys**, click **Create**.
  5. Enter some **Name** and an optional **Description** for the API key.
  6. Leave **Leaked action** set to **Disable the leaked key** and **Session management** set to **No**.
  7. Click **Create**.
  8. Click **Copy** or **Download** to copy or save the API key to a secure location. You won't be able to access this API key from this screen again.

* An IBM Cloud Object Storage (COS) instance in the account, and a bucket within that instance. If you do not have them already,
  create them as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. Click **Create resource**.
  4. With **IBM Cloud catalog** selected, search for and select **Object Storage**.
  5. Complete the on-screen instructions to finish creating the COS instance.
  6. With the COS instance's settings page shown, on the **Buckets** tab, click **Create bucket**.
  7. Complete the on-screen instructions to finish creating the bucket.

* The name, region, and public endpoint for the target bucket within the target Cloud Object Storage (COS) instance. To get these:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Storage**, and then click the target COS instance.
  4. On the **Buckets** tab, click the target bucket.
  5. On the **Configuration** tab, note the following:

     * Under **Bucket details**, note the **Bucket name**. This is the bucket's name.
     * Under **Bucket details** section, note the value inside of the parentheses inside **Location**, for example `us-east`. This is the bucket's region.
     * Under **Endpoints**, note the value of **Public**, for example `s3.us-east.cloud-object-storage.appdomain.cloud`. (Ignore the values of
       **Private** and **Direct**). This is the bucket's public endpoint.

* An HMAC access key ID and secret access key for the target Cloud Object Storage (COS) instance. If you do not have them already,
  get or create them as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.

  3. In the list of resources, expand **Storage**, and then click the target COS instance.

  4. On the **Service credentials** tab, if there is a credential that you want to use in the list, expand the credential, and copy the following values to a secure location:

     * `access_key_id` under `cos_hmac_keys`, which represents the HMAC access key ID.
     * `secret_access_key` under `cos_hmac_keys`, which represents the HMAC secret access key.

     After you have copied the preceding values, you have completed this procedure.

  5. If there is not a credential that you want to use, or there are no credentials at all, click **New Credential**.

  6. Enter some **Name** for the credential.

  7. For **Role**, select at least **Writer**, leave **Select Service ID** set to **Auto Generated**,
     switch on **Include HMAC Credential**, and then click **Add**.

  8. In the list of credentials, expand the credential, and copy the following values to a secure location:

     * `access_key_id` under `cos_hmac_keys`, which represents the HMAC access key ID.
     * `secret_access_key` under `cos_hmac_keys`, which represents the HMAC secret access key.

* An IBM watsonx.data data store instance in the IBM Cloud account. If you do not have one already, create one as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. Click **Create resource**.
  4. With **IBM Cloud catalog** selected, search for and select **watsonx.data**.
  5. Complete the on-screen instructions to finish creating the watsonx.data data store instance.

* An Apache Iceberg-based catalog within the watsonx.data data store instance. If you do not have one already, create one as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.

  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.

  4. Click **Open web console**.

  5. If prompted, log in to the web console.

  6. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.

  7. Click **Add component**.

  8. Under **Storage**, click **IBM Cloud Object Storage**, and then click **Next**.

  9. Complete the on-screen instructions to finish creating the Iceberg catalog. This includes providing the following settings:

     If you select **Discover COS instance**, you must provide the following settings:

     * The name of the target COS instance.
     * The name of the target storage object (such as the target bucket) within the COS instance.
     * Some display name for the component.
     * After you provide this information, do the following:

       a. Check the box labelled **Associate Catalog**.<br />
       b. For **Catalog type**, select **Apache Iceberg**.<br />
       c. Enter some **Catalog name**.<br />
       d. Click **Associate**.<br />

     If you select **Register my own**, you must provide the following settings:

     * Some display name for the component.
     * The name of the target bucket within the target Cloud Object Storage (COS) instance that you noted earlier.
     * The region for the target bucket, which you noted earlier.
     * The public endpoint for the target bucket, which you noted earlier. For this screen only, be sure to prefix the public endpoint with `https://`.
     * The HMAC access key ID for the target COS instance, which you noted earlier.
     * The HMAC secret access key for the target COS instance, which you noted earlier.
     * After you provide this information, do the following:

       a. Check the box labelled **Associate Catalog**.<br />
       b. For **Catalog type**, select **Apache Iceberg**.<br />
       c. Enter some **Catalog name**.<br />
       d. Click **Associate**.<br />

  10. On the sidebar, click **Infrastructure manager**. Make sure the catalog is associated with the appropriate engines. If it is not, rest your mouse
      on an unassociated target engine, click the **Manage associations** icon, check the box next to the target catalog's name, and then
      click **Save and restart engine**.

      To create an engine if one is not already shown, click **Add component**, and follow the on-screen to add an appropriate engine from the list of available **Engines**
      (for example, an **IBM Presto** engine).

* The catalog name and metastore REST endpoint for the target Iceberg catalog. To get this:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. In the **Catalogs** section, click the target Iceberg catalog.
  8. On the **Details** tab, note the value of **Name** representing the catalog name, and **Metastore REST endpoint** representing the metastore REST endpoint. (Ignore the **Metastore Thrift endpoint** value.)

* A namespace (also known as a schema) and a table in the target catalog. If you do not have these already, create them as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Data manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. On the **Browse data** tab, under **Catalogs associated**, click the target catalog.
  8. Click the ellipses, and then click **Create schema**.
  9. Enter some **Name** for the schema, and then click **Create**.
  10. On the sidebar, click **Query workspace**.
  11. In the SQL editor, enter and run a table creation statement such as the following one that uses
      [Presto SQL](https://prestodb.io/docs/current/connector/iceberg.html) syntax, replacing `<catalog-name>` with the name of the target
      catalog and `<schema-name>` with the name of the target schema, with at minimum the fields `id`, `type`, `text`, `element_id`, and `record_id`:

      ```sql theme={null}
      CREATE TABLE <catalog-name>.<schema-name>.elements (
         id varchar,
         type varchar,
         text varchar,
         element_id varchar,
         record_id varchar,
         <additional-fields-as-needed>
      )
      WITH (
         delete_mode = 'copy-on-write',
         format = 'PARQUET',
         format_version = '2'
      )
      ```

      <Info>
        The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
      </Info>

      For objects in the `metadata` field that Unstructured produces and that you want to store in an Apache Iceberg table in IBM watsonx.data,
      you must create fields in your table schema that follows Unstructured's `metadata` field naming convention. For example, if Unstructured
      produces a `metadata` field with the following child objects:

      ```json theme={null}
      "metadata": {
        "is_extracted": "true",
        "coordinates": {
          "points": [
            [
              134.20055555555555,
              241.36027777777795
            ],
            [
              134.20055555555555,
              420.0269444444447
            ],
            [
              529.7005555555555,
              420.0269444444447
            ],
            [
              529.7005555555555,
              241.36027777777795
            ]
          ],
          "system": "PixelSpace",
          "layout_width": 1654,
          "layout_height": 2339
        },
        "filetype": "application/pdf",
        "languages": [
          "eng"
        ],
        "page_number": 1,
        "image_base64": "aaa...bbb...111",
        "image_mime_type": "image/jpeg",
        "filename": "realestate.pdf",
        "data_source": {
          "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
          "record_locator": {
            "protocol": "file",
            "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
          }
        },
        "entities": {
          "items": [
            {
              "entity": "HOME FOR FUTURE",
              "type": "ORGANIZATION"
            },
            {
              "entity": "221 Queen Street, Melbourne VIC 3000",
              "type": "LOCATION"
            }
          ],
          "relationships": [
            {
              "from": "HOME FOR FUTURE",
              "relationship": "based_in",
              "to": "221 Queen Street, Melbourne VIC 3000"
            }
          ]
        }
      }
      ```

      You could create corresponding fields in your table schema with the following field names and data types:

      | **Field name**    | **Apache Iceberg data type**                                                                                                        |
      | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
      | `is_extracted`    | `varchar`                                                                                                                           |
      | `points`          | `varchar`                                                                                                                           |
      | `system`          | `varchar`                                                                                                                           |
      | `layout_width`    | `bigint`                                                                                                                            |
      | `layout_height`   | `bigint`                                                                                                                            |
      | `filetype`        | `varchar`                                                                                                                           |
      | `languages`       | `array(varchar)`                                                                                                                    |
      | `page_number`     | `varchar`                                                                                                                           |
      | `image_base64`    | `varchar`                                                                                                                           |
      | `image_mime_type` | `varchar`                                                                                                                           |
      | `filename`        | `varchar`                                                                                                                           |
      | `url`             | `varchar`                                                                                                                           |
      | `record_locator`  | `varchar`                                                                                                                           |
      | `entities`        | `row(items array(row(entity varchar, type varchar)), relationships array(row("from" varchar, relationship varchar, "to" varchar)))` |

      Incoming elements that do not have matching column
      names will be dropped upon record insertion. For example, if the incoming data has an element named `sent_from` and there is no
      column named `sent_from` in the table, the `sent_from` element will be dropped upon record insertion. You should modify the preceding
      sample table creation statement to add columns for any additional elements that you want to be included upon record
      insertion.

      To increase query performance, Iceberg uses [hidden partitioning](https://iceberg.apache.org/docs/latest/partitioning/) to
      group similar rows together when writing. You can also
      [explicitly define partitions](https://prestodb.io/docs/current/connector/iceberg.html#create-table) as part of the
      preceding `CREATE TABLE` statement.

* The name of the target namespace (also known as a schema) within the target catalog, and name of the target table within that schema. To get these:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Data manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. On the **Browse data** tab, expand the name of the target catalog, and note the names of the target schema and target table.

* The name of the column in the target table that uniquely identifies each of the records in the table.

* To improve performance, the target table should be set to regularly remove old metadata files. To do this, run the following Python script.
  (You cannot use the preceding `CREATE TABLE` statement, or other SQL statements such as `ALTER TABLE`, to set this behavior.) To get the
  values for the specified environment variables, see the preceding instructions.

  ```python theme={null}
  # Improves performance by setting the target table to regularly remove 
  # old metadata files. 
  #
  # First, install the following dependencies into your Python virtual 
  # environment:
  # 
  # pip install requests pyiceberg pyarrow
  #
  # Then, set the following environment variables:
  #
  # IBM_IAM_API_KEY - An API key value for the target IBM Cloud account.
  # IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT - The metastore REST endpoint 
  #     value for the target Apache Iceberg catalog in the target IBM watsonx.data 
  #     data store instance.
  # IBM_COS_BUCKET_PUBLIC_ENDPOINT - The target IBM Cloud Object Storage (COS) 
  #     instance’s endpoint value.
  # IBM_COS_ACCESS_KEY - An HMAC access key ID for the target COS instance.
  # IBM_COS_SECRET_ACCESS_KEY - The associated HMAC secret access key ID for the 
  #     target HMAC access key.
  # IBM_COS_BUCKET_REGION - The target COS instance’s region short ID.
  # IBM_ICEBERG_CATALOG - The name of the target Iceberg catalog.
  # IBM_ICEBERG_SCHEMA - The name of the target namespace (also known as a schema) 
  #     in the target catalog.
  # IBM_ICEBERG_TABLE - The name of the target table in the target schema.
  #
  # To get these values, see the Unstructured documentation for the 
  #     IBM watsonx.data connector.

  import os
  import requests
  from pyiceberg.catalog import load_catalog

  def main():
     # Get a bearer token for the target IBM Cloud account.   
     bearer_token = requests.post(
        url="https://iam.cloud.ibm.com/identity/token",
        headers={
              "Content-Type": "application/x-www-form-urlencoded",
              "Accept": "application/json"
        },
        data={
              "grant_type": "urn:ibm:params:oauth:grant-type:apikey", 
              "apikey": os.getenv("IBM_IAM_API_KEY")
        }
     ).json().get("access_token")

     # Connect to the target Iceberg catalog.
     catalog = load_catalog(
        os.getenv("IBM_ICEBERG_CATALOG"),
        **{
              "type": "rest",
              "uri": f"https://{os.getenv("IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT")}/mds/iceberg",
              "token": bearer_token,
              "warehouse": os.getenv("IBM_ICEBERG_CATALOG"),
              "s3.endpoint": os.getenv("IBM_COS_BUCKET_PUBLIC_ENDPOINT"),
              "s3.access-key-id": os.getenv("IBM_COS_ACCESS_KEY"),
              "s3.secret-access-key": os.getenv("IBM_COS_SECRET_ACCESS_KEY"),
              "s3.region": os.getenv("IBM_COS_BUCKET_REGION")
        },
     )
              
     # Load the target table.
     table = catalog.load_table(f"{os.getenv("IBM_ICEBERG_SCHEMA")}.{os.getenv("IBM_ICEBERG_TABLE")}")

     # Set the target table's properties to remove old metadata files.
     with table.transaction() as transaction:
        transaction.set_properties(
              {
                 "commit.manifest.min-count-to-merge": 10,
                 "commit.manifest-merge.enabled": True,
                 "write.metadata.previous-versions-max": 10,
                 "write.metadata.delete-after-commit.enabled": True,
              }
        )

     # Confirm that the target table's properties were set as expected.
     print(table.metadata.properties)

  if __name__ == "__main__":
     main()
  ```

The IBM watsonx.data connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[ibm-watsonx-s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `IBM_IAM_API_KEY` - An API key for the target IBM Cloud account, represented by `--iam-api-key` (CLI) or `iam_api_key` (Python).
* `IBM_COS_ACCESS_KEY` - An HMAC access key ID for the target IBM Cloud Object Storage (COS) instance, represented by `--access-key-id` (CLI) or `access_key_id` (Python).
* `IBM_COS_SECRET_ACCESS_KEY` - The associated HMAC secret access key ID for the target HMAC access key, represented by `--secret-access-key` (CLI) or `secret_access_key` (Python).
* `IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT` - The metastore REST endpoint value for the target Apache Iceberg catalog in the target IBM watsonx.data data store instance, represented by `--iceberg-endpoint` (CLI) or `iceberg_endpoint` (Python). Do not include `https://` in this value.
* `IBM_COS_BUCKET_PUBLIC_ENDPOINT` - The target COS instance's endpoint value, represented by `--object-storage-endpoint` (CLI) or `object_storage_endpoint` (Python).
* `IBM_COS_BUCKET_REGION` - The target COS instance's region short ID, represented by `--object-storage-region` (CLI) or `object_storage_region` (Python).
* `IBM_ICEBERG_CATALOG` - The name of the target Iceberg catalog, represented by `--catalog` (CLI) or `catalog` (Python).
* `IBM_ICEBERG_SCHEMA` - The name of the target namespace (also known as a schema) in the target catalog, represented by `--namespace` (CLI) or `namespace` (Python).
* `IBM_ICEBERG_TABLE` - The name of the target table in the target schema, represented by `--table` (CLI) or `table` (Python).
* `IBM_ICEBERG_TABLE_UNIQUE_RECORD_COLUMN` - The name of the column that uniquely identifies each record in the target table, represented by `--record-id-key` (CLI) or `record_id_key` (Python). The default is `record_id`.

Additionally:

* `--max-retries-connection` (CLI) or `max_retries_connection` (Python) is an optional parameter that specifies the maximum number of retries when connecting to the catalog. Typically, an optimal setting is `15`. The default is `10`. If specified, it must be a number between `2` and `100`, inclusive.
* `--max-retries` (CLI) or `max_retries` (Python) is an optional parameter that specifies the number of times to retry uploading data. Typically, an optimal setting is `150`. The default is `50`. If specified, it must be a number between `2` and `500`, inclusive.

Now call the Unstructured Ingest CLI or Unstructured Ingest Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    ibm-watsonx-s3 \
      --iam-api-key $IBM_IAM_API_KEY \
      --access-key-id $IBM_COS_ACCESS_KEY \
      --secret-access-key $IBM_COS_SECRET_ACCESS_KEY \
      --iceberg-endpoint $IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT \
      --object-storage-endpoint $IBM_COS_BUCKET_PUBLIC_ENDPOINT \
      --object-storage-region $IBM_COS_BUCKET_REGION \
      --catalog $IBM_ICEBERG_CATALOG \
      --namespace $IBM_ICEBERG_SCHEMA \
      --table $IBM_ICEBERG_TABLE \
      --max-retries 50 \
      --max-retries-connection 10 \
      --record-id-key $IBM_ICEBERG_TABLE_UNIQUE_RECORD_COLUMN
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.processes.connectors.ibm_watsonx.ibm_watsonx_s3 import (
      IbmWatsonxConnectionConfig,
      IbmWatsonxAccessConfig,
      IbmWatsonxUploadStagerConfig,
      IbmWatsonxUploaderConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(reprocess=True, clear_cache=True),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=IbmWatsonxConnectionConfig(
              access_config=IbmWatsonxAccessConfig(
                  iam_api_key=os.getenv("IBM_IAM_API_KEY"),
                  access_key_id=os.getenv("IBM_COS_ACCESS_KEY"),
                  secret_access_key=os.getenv("IBM_COS_SECRET_ACCESS_KEY")
              ),
              iceberg_endpoint=os.getenv("IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT"),
              object_storage_endpoint=os.getenv("IBM_COS_BUCKET_PUBLIC_ENDPOINT"),
              object_storage_region=os.getenv("IBM_COS_BUCKET_REGION"),
              catalog=os.getenv("IBM_ICEBERG_CATALOG")
          ),
          stager_config=IbmWatsonxUploadStagerConfig(),
          uploader_config=IbmWatsonxUploaderConfig(
              namespace=os.getenv("IBM_ICEBERG_SCHEMA"),
              table=os.getenv("IBM_ICEBERG_TABLE"),
              max_retries=50,
              max_retries_connection=10,
              record_id_key=os.getenv("IBM_ICEBERG_TABLE_UNIQUE_RECORD_COLUMN")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Kafka
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/kafka



Batch process all your records to store structured outputs in Kafka.

The requirements are as follows.

* A Kafka cluster in [Confluent Cloud](https://www.confluent.io/confluent-cloud).
  ([Create a cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#create-ak-clusters).)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

  <iframe title="YouTube video player" />

* The [hostname and port number](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#view-a-ak-cluster) of the bootstrap Kafka cluster to connect to..

* The name of the topic to read messages from or write messages to on the cluster.
  [Create a topic](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).
  [Access available topics](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).

* For authentication, an [API key and secret](https://docs.confluent.io/cloud/current/security/authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.html#add-an-api-key).

The Kafka connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[kafka]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `KAFKA_BOOTSTRAP_SERVER` - The hostname of the bootstrap Kafka cluster to connect to, represented by `--bootstrap-server` (CLI) or `bootstrap_server` (Python).
* `KAFKA_PORT` - The port number of the cluster, represented by `--port` (CLI) or `port` (Python).
* `KAFKA_TOPIC` - The unique name of the topic to read messages from and write messages to on the cluster, represented by `--topic` (CLI) or `topic` (Python).

If you use Kafka API keys and secrets for authentication:

* `KAFKA_API_KEY` - The Kafka API key value, represented by `--kafka-api-key` (CLI) or `kafka_api_key` (Python).
* `KAFKA_SECRET` - The secret value for the Kafka API key, represented by `--secret` (CLI) or `secret` (Python).

Additional settings include:

* `--confluent` (CLI) or `confluent` (Python): True to indicate that the cluster is running Confluent Kafka.
* `--num-messages-to-consume` (CLI) or `num_messages_to_consume` (Python): The maximum number of messages to get from the topic. The default is `1` if not otherwise specified.
* `--timeout` (CLI) or `timeout` (Python): The maximum amount of time to wait for the response of a request to the topic, expressed in seconds. The default is `1.0` if not otherwise specified.
* `--group-id` (CLI) or `group_id` (Python): The ID of the consumer group, if any, that is associated with the target Kafka cluster.
  (A consumer group is a way to allow a pool of consumers to divide the consumption of data
  over topics and partitions.) The default is `default_group_id` if not otherwise specified.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY\
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    kafka \
      --bootstrap-server $KAFKA_BOOTSTRAP_SERVER \ 
      --port $KAFKA_PORT \
      --topic $KAFKA_TOPIC \
      --kafka-api-key $KAFKA_API_KEY \
      --secret $KAFKA_API_KEY \
      --confluent true \
      --num-messages-to-consume 1 \
      --timeout 1.0
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.kafka.cloud import (
      CloudKafkaConnectionConfig,
      CloudKafkaAccessConfig,
      CloudKafkaUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=CloudKafkaConnectionConfig(
              access_config=CloudKafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET")
              ),
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT")
          ),
          uploader_config=CloudKafkaUploaderConfig(
              batch_size=100,
              topic=os.getenv("KAFKA_TOPIC"),
              timeout=10
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# KDB.AI
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/kdbai



Batch process all your records to store structured outputs in KDB.AI.

The requirements are as follows.

* A KDB.AI Cloud or server instance. [Sign Up for KDB.AI Cloud: Starter Edition](https://trykdb.kx.com/kdbai/signup/). [Set up KDB.AI Server](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html).
* The instance's endpoint URL. [Get the KDB.AI Cloud endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#connect-to-your-database). [Get the KDB.AI Server endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html).
* An API key. [Create the API key](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#create-an-api-key).
* The name of the target table to access. [Create the table](https://code.kx.com/kdbai/latest/gettingStarted/quickstart.html#create-a-new-table).

  KDB.AI requires the target table to have a defined schema before Unstructured can write to the table. The recommended table
  schema for Unstructured contains the fields `id`, `element_id`, `document`, `metadata`, and `embeddings`, as follows.
  This example code demonstrates the use of the [KDB.AI Client for Python](https://pypi.org/project/kdbai-client/) to create
  a table with this recommended schema, along with creating a vector index that contains 3072 dimensions:

  ```python Python theme={null}
  import kdbai_client as kdbai
  import os

  session = kdbai.Session(
      endpoint=os.getenv("KDBAI_ENDPOINT"),
      api_key=os.getenv("KDBAI_API_KEY")
  )

  db = session.database("default")

  schema = [
      {
          "name": "id",
          "type": "str"
      },
      {
          "name": "element_id",
          "type": "str"
      },
      {
          "name": "document",
          "type": "str"
      },
      {
          "name": "metadata", 
          "type": "general"
      },
      {
          "name": "embeddings",
          "type": "float32s"
      }
  ]

  indexes = [ 
      {
          "name": "vectorIndex",
          "type": "flat", 
          "params": {
              "dims": 3072,
              "metric": "CS"
          },
          "column": "embeddings"
      }
  ]

  table = db.create_table(
      table=os.getenv("KDBAI_TABLE"),
      schema=schema,
      indexes=indexes
  )

  print(f"The table named '{table.name}' now exists.")
  ```

The KDB.AI connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[kdbai]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `KDBAI_ENDPOINT` - The KDB.AI instance's endpoint URL, represented by `--endpoint` (CLI) or `endpoint` (Python).
* `KDBAI_API_KEY` - The KDB.AI API key, represented by `--api-key` (CLI) or `api_key` (Python).
* `KDBAI_TABLE` - The name of the target table, represented by `--table-name` (CLI) or `table_name` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    kdbai \
      --endpoint $KDBAI_API_KEY \
      --api-key $KDBAI_API_KEY \
      --table-name $KDBAI_TABLE
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.kdbai import (
      KdbaiConnectionConfig,
      KdbaiAccessConfig,
      KdbaiUploadStagerConfig,
      KdbaiUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=KdbaiConnectionConfig(
              access_config=KdbaiAccessConfig(
                  api_key=os.getenv("KDBAI_API_KEY")
              ),
              endpoint=os.getenv("KDBAI_ENDPOINT")
          ),
          stager_config=KdbaiUploadStagerConfig(),
          uploader_config=KdbaiUploaderConfig(table_name=os.getenv("KDBAI_TABLE"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# LanceDB
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/lancedb



Batch process all your records to store structured outputs in LanceDB.

The requirements are as follows.

* A [LanceDB open source software (OSS) installation](https://lancedb.github.io/lancedb/basic/#installation) on a local machine, a server, or a virtual machine.
  (LanceDB Cloud is not supported.)

* For LanceDB OSS with local data storage:

  * The local path to the folder where the LanceDB data is (or will be) stored.
    See [Connect to a database](https://lancedb.github.io/lancedb/basic/#connect-to-a-database) in the LanceDB documentation.
  * The name of the target [LanceDB table](https://lancedb.github.io/lancedb/basic/#create-an-empty-table) within the local data folder.

    <Note>
      Unstructured recommends that the target table have a field named `record_id` with a text string data type.
      Unstructured can use this field to do intelligent record overwrites. Without this field, duplicate records
      might be written to the table or, in some cases, the operation could fail altogether.
    </Note>

* For LanceDB OSS with data storage in an Amazon S3 bucket:

  * The URI for the target Amazon S3 bucket and any target folder path within that bucket. Use the format `s3://<bucket-name>[/<folder-name>]`.

  * The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Amazon S3 bucket.

    <Note>
      Unstructured recommends that the target table have a field named `record_id` with a text string data type.
      Unstructured can use this field to do intelligent record overwrites. Without this field, duplicate records
      might be written to the table or, in some cases, the operation could fail altogether.
    </Note>

  * The AWS access key ID and AWS secret access key for the AWS IAM entity that has access to the Amazon S3 bucket.

  For more information, see [AWS S3](https://lancedb.github.io/lancedb/guides/storage/#aws-s3) in the LanceDB documentation, along with the following video:

  <iframe title="YouTube video player" />

* For LanceDB OSS with data storage in an Azure Blob Storage account:

  * The name of the target Azure Blob Storage account.
    \= The URI for the target container within that Azure Blob Storage account and any target folder path within that container. Use the format `az://<container-name>[/<folder-name>]`.

  * The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Azure Blob Storage account.

    <Note>
      Unstructured recommends that the target table have a field named `record_id` with a text string data type.
      Unstructured can use this field to do intelligent record overwrites. Without this field, duplicate records
      might be written to the table or, in some cases, the operation could fail altogether.
    </Note>

  * The access key for the Azure Blob Storage account.

  For more information, see [Azure Blob Storage](https://lancedb.github.io/lancedb/guides/storage/#azure-blob-storage) in the LanceDB documentation, along with the following video:

  <iframe title="YouTube video player" />

* For LanceDB OSS with data storage in a Google Cloud Storage bucket:

  * The URI for the target Google Cloud Storage bucket and any target folder path within that bucket. Use the format `gs://<bucket-name>[/<folder-name>]`.

  * The name of the target [LanceDB table](https://lancedb.github.io/lancedb/guides/storage/#object-stores) within the Google Cloud Storage bucket.

    <Note>
      Unstructured recommends that the target table have a field named `record_id` with a text string data type.
      Unstructured can use this field to do intelligent record overwrites. Without this field, duplicate records
      might be written to the table or, in some cases, the operation could fail altogether.
    </Note>

  * A single-line string that contains the contents of the downloaded service account key file for the Google Cloud service account that has access to the
    Google Cloud Storage bucket.

  For more information, see [Google Cloud Storage](https://lancedb.github.io/lancedb/guides/storage/#google-cloud-storage) in the LanceDB documentation, along with the following video:

  <iframe title="YouTube video player" />

The LanceDB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[lancedb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* For LanceDB OSS with local data storage:

  * `LANCEDB_URI` - The local path to the folder where the LanceDB data is stored, represented by `--uri` (CLI) or `uri` (Python).
  * `LANCEDB_TABLE` - The name of the target LanceDB table within the local data folder, represented by `--table-name` (CLI) or `table_name` (Python).

* For LanceDB OSS with data storage in an Amazon S3 bucket:

  * `LANCEDB_URI` - The URI for the target Amazon S3 bucket and any target folder path within that bucket. Use the format `s3://<bucket-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
  * `LANCEDB_TABLE` - The name of the target LanceDB table within the Amazon S3 bucket, rrepresented by `--table-name` (CLI) or `table_name` (Python).
  * `AWS_ACCESS_KEY_ID` - The AWS access key ID for the AWS IAM entity that has access to the Amazon S3 bucket, represented by `--aws-access-key-id` (CLI) or `aws_access_key_id` (Python).
  * `AWS_SECRET_ACCESS_KEY` - The AWS secret access key for the AWS IAM entity that has access to the Amazon S3 bucket, represented by `--aws-secret-access-key` (CLI) or `aws_secret_access_key` (Python).

* For LanceDB OSS with data storage in an Azure Blob Storage account:

  * `LANCEDB_URI` - The URI for the target container within that Azure Blob Storage account and any target folder path within that container. Use the format `az://<container-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
  * `LANCEDB_TABLE` - The name of the target LanceDB table within the Azure Blob Storage account, represented by `--table-name` (CLI) or `table_name` (Python).
  * `AZURE_STORAGE_ACCOUNT_NAME` - The name of the target Azure Blob Storage account, represented by `--azure-storage-account-name` (CLI) or `azure_storage_account_name` (Python).
  * `AZURE_STORAGE_ACCOUNT_KEY` - The access key for the Azure Blob Storage account, represented by `--azure-storage-account-key` (CLI) or `azure_storage_account_key` (Python).

* For LanceDB OSS with data storage in a Google Cloud Storage bucket:

  * `LANCEDB_URI` - The URI for the target Google Cloud Storage bucket and any target folder path within that bucket. Use the format `gs://<bucket-name>[/<folder-name>]`. This is represented by `--uri` (CLI) or `uri` (Python).
  * `LANCEDB_TABLE` - The name of the target LanceDB table within the Google Cloud Storage bucket, represented by `--table-name` (CLI) or `table_name` (Python).
  * `GCS_SERVICE_ACCOUNT_KEY` - A single-line string that contains the contents of the downloaded service account key file for the Google Cloud service account
    that has access to the Google Cloud Storage bucket, represented by `--google-service-account-key` (CLI) or `google_service_account_key` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For LanceDB OSS with local data storage:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-local \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE

  # For LanceDB OSS with data storage in an Amazon S3 bucket:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-aws \
      --aws-access-key-id $AWS_ACCESS_KEY_ID \
      --aws-secret-access-key $AWS_SECRET_ACCESS_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s

  # For LanceDB OSS with data storage in an Azure Blob Storage account:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-azure \
      --azure-storage-account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --azure-storage-account-key $AZURE_STORAGE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s

  # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    lancedb-gcs \
      --google-service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --uri $LANCEDB_URI \
      --table-name $LANCEDB_TABLE \
      --timeout 30s
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # For LanceDB OSS with local data storage:
  # from unstructured_ingest.processes.connectors.lancedb.local import (
  #     LanceDBLocalConnectionConfig,
  #     LanceDBLocalAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # For LanceDB OSS with data storage in an Amazon S3 bucket:
  from unstructured_ingest.processes.connectors.lancedb.aws import (
      LanceDBS3ConnectionConfig,
      LanceDBS3AccessConfig,
      LanceDBUploadStagerConfig,
      LanceDBUploaderConfig
  )

  # For LanceDB OSS with data storage in an Azure Blob Storage account:
  # from unstructured_ingest.processes.connectors.lancedb.azure import (
  #     LanceDBAzureConnectionConfig,
  #     LanceDBAzureAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
  # from unstructured_ingest.processes.connectors.lancedb.gcp import (
  #     LanceDBGCSConnectionConfig,
  #     LanceDBGCSAccessConfig,
  #     LanceDBUploadStagerConfig,
  #     LanceDBUploaderConfig
  # )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),

          # For LanceDB OSS with local data storage:
          # destination_connection_config=LanceDBLocalConnectionConfig(
          #     access_config=LanceDBLocalAccessConfig(),
          #     uri=os.getenv("LANCEDB_URI")
          # ),

          # For LanceDB OSS with data storage in an Amazon S3 bucket:
          destination_connection_config=LanceDBS3ConnectionConfig(
              access_config=LanceDBS3AccessConfig(
                  aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
                  aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
              ),
              uri=os.getenv("LANCEDB_URI"),
              timeout="30s"
          ),

          # For LanceDB OSS with data storage in an Azure Blob Storage account:
          # destination_connection_config=LanceDBAzureConnectionConfig(
          #     access_config=LanceDBAzureAccessConfig(
          #         azure_storage_account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
          #         azure_storage_account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
          #     ),
          #     uri=os.getenv("LANCEDB_URI"),
          #     timeout="30s"
          # ),

          # For LanceDB OSS with data storage in a Google Cloud Storage bucket:
          # destination_connection_config=LanceDBGCSConnectionConfig(
          #     access_config=LanceDBGCSAccessConfig(
          #         google_service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
          #     ),
          #     uri=os.getenv("LANCEDB_URI"),
          #     timeout="30s"
          # ),

          stager_config=LanceDBUploadStagerConfig(),
          uploader_config=LanceDBUploaderConfig(table_name=os.getenv("LANCEDB_TABLE"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Local
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/local



Batch process all your records to store structured outputs locally.

You will need the local destination connector dependencies:

```bash CLI, Python theme={null}
pip install unstructured-ingest
```

To use the local destination connector, you must set `--output-dir` (CLI) or `output_dir` (Python) to the path in the local filesystem which will contain the structured output.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library.  The source connector can be any of the ones supported.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured for processing:

<iframe title="YouTube video player" />

This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Milvus
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/milvus



Batch process all your records to store structured outputs in Milvus.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Milvus cloud-based instances (such as Milvus on IBM watsonx.data, or Zilliz Cloud) are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Milvus local and cloud-based instances are supported.

* For Milvus on IBM watsonx.data, you will need:

  <iframe title="YouTube video player" />

  * An [IBM Cloud account](https://cloud.ibm.com/registration).

  * An IBM watsonx.data [Lite plan](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_prov_lite_1)
    or [Enterprise plan](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started_1) within your IBM Cloud account.

    * If you are provisioning a Lite plan, be sure to choose the **Generative AI** use case when prompted, as this is the only use case offered that includes Milvus.

  * A [Milvus service instance in IBM watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-adding-milvus-service).

    * If you are creating a Milvus service instance within a watsonx.data Lite plan, when you are prompted to choose a Milvus instance size, you can only select **Lite**. Because the Lite
      Milvus instance size is recommended only for 384 dimensions, you should also use an embedding model that uses 384 dimensions only.
    * If you are creating a Milvus service instance within a watsonx.data Enterprise plan, you can choose any available Milvus instance size. However, all Milvus instance sizes other than
      **Custom** are recommended only for 384 dimensions, which means you should use an embedding model that uses 384 dimensions only.
      The **Custom** Milvus instance size is recommended for any number of dimensions.

  * The URI of the instance, which takes the format of `https://`, followed by the instance's **GRPC host**, followed by a colon and the **GRPC port**.
    This takes the format of `https://<host>:<port>`. To get this information, do the following:

    a. Sign in to your IBM Cloud account.<br />
    b. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the title bar.<br />
    c. Expand **Databases**, and then click the name of the target **watsonx.data** plan.<br />
    d. Click **Open web console**.<br />
    e. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the title bar.<br />
    f. Click the target Milvus service instance.<br />
    g. On the **Details** tab, under **Type**, click **View connect details**.<br />
    h. Under **Service details**, expand **GRPC**, and note the value of **GRPC host** and **GRPC port**.<br />

  * The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.

  * The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database. Note the collection requirements at the end of this section.

  * The username and password to access the instance.

    * The username for Milvus on IBM watsonx.data is typically `ibmlhapikey`.

      <Note>
        More recent versions of Milvus on IBM watsonx.data require `ibmlhapikey_<your-IBMid>` instead, where `<your-IBMid>` is
        your IBMid, for example `me@example.com`. To get your IBMid, do the following:

        1. Sign in to your IBM Cloud account.
        2. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
        3. In the sidebar, expand **Manage identities**, and then click **Users**.
        4. In the list of users, click your user name.
        5. On the **User details** tab, in the **Details** tile, note the value of **IBMid**.
      </Note>

    * The password for Milvus on IBM watsonx.data is in the form of an IBM Cloud user API key. To create an IBM Cloud user API key:

      a. Sign in to your IBM Cloud account.<br />
      b. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.<br />
      c. On the sidebar, under **Manage identities**, click **API keys**. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the title bar.<br />
      d. Click **Create**.<br />
      e. Enter some **Name** for the API key.<br />
      f. Optionally, enter some **Description** for the API key.<br />
      g. For **Leaked action**, leave **Disable the leaked key** selected.<br />
      h. For **Session management**, leave **No** selected.<br />
      i. Click **Create**.<br />
      j. Click **Download** (or **Copy**), and then download the API key to a secure location (or paste the copied API key into a secure location). You won't be able to access this API key from this dialog again. If you lose this API key, you can create a new one (and you should then delete the old one).<br />

* For Zilliz Cloud, you will need:

  <iframe title="YouTube video player" />

  * A [Zilliz Cloud account](https://cloud.zilliz.com/signup).

  * A [Zilliz Cloud cluster](https://docs.zilliz.com/docs/create-cluster).

  * The URI of the cluster, also known as the cluster's *public endpoint*, which takes a format such as
    `https://<cluster-id>.<cluster-type>.<cloud-provider>-<region>.cloud.zilliz.com`. To get this public endpoint value, do the following:

    1. After you sign in to your Zilliz Cloud account, on the sidebar, in the list of available projects, select the project that contains the cluster.
    2. On the sidebar, click **Clusters**.
    3. Click the tile for the cluster.
    4. On the **Cluster Details** tab, on the **Connect** subtab, copy the **Public Endpoint** value.

  * The username and password to access the cluster, as follows:

    1. After you sign in to your Zilliz Cloud account, on the sidebar, in the list of available projects, select the project that contains the cluster.
    2. On the sidebar, click **Clusters**.
    3. Click the tile for the cluster.
    4. On the **Users** tab, copy the name of the user.
    5. Next to the user's name, under **Actions**, click the ellipsis (three dots) icon, and then click **Reset Password**.
    6. Enter a new password for the user, and then click **Confirm**. Copy this new password.

  * The name of the [database](https://docs.zilliz.com/docs/database#create-database) in the instance.

  * The name of the [collection](https://docs.zilliz.com/docs/manage-collections-console#create-collection) in the database.

    The collection must have a defined schema before Unstructured can write to the collection. The minimum viable
    schema for Unstructured contains only the fields `element_id`, `embeddings`, `record_id`, and `text`, as follows.
    `type` is an optional field, but highly recommended. For settings for additional Unstructured-produced fields,
    such as the ones within `metadata`, see the usage notes toward the end of this section and adapt them to your specific needs.

    | Field Name                       | Field Type        | Max Length | Dimension |
    | -------------------------------- | ----------------- | ---------- | --------- |
    | `element_id` (primary key field) | **VARCHAR**       | `200`      | --        |
    | `embeddings` (vector field)      | **FLOAT\_VECTOR** | --         | `384`     |
    | `record_id`                      | **VARCHAR**       | `200`      | --        |
    | `text`                           | **VARCHAR**       | `65536`    | --        |
    | `type`                           | **VARCHAR**       | `200`      | --        |

    In the **Create Index** area for the collection, next to **Vector Fields**, click **Edit Index**. Make sure that for the
    `embeddings` field, the **Field Type** is set to **FLOAT\_VECTOR** and the **Metric Type** is set to **Cosine**.

    <Warning>
      The number of dimensions for the `embeddings` field must match the number of dimensions for the embedding model that you plan to use.
    </Warning>

    <Note>
      Fields with a **VARCHAR** data type are limited to a maximum length of 65,535 characters. Attempting to exceed this character count
      will cause Unstructured to throw errors when attempting to write to a Milvus collection, and the associated Unstructured job could fail.
      For example, `metadata` fields that typically exceed these character counts include `image_base64` and `orig_elements`.
    </Note>

    <Info>
      The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
    </Info>

* For Milvus local, you will need:

  * A [Milvus instance](https://milvus.io/docs/install-overview.md).
  * The [URI](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of the instance.
  * The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.
  * The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database.
    Note the collection requirements at the end of this section.
  * The [username and password, or token](https://milvus.io/docs/authenticate.md) to access the instance.

All Milvus instances require the target collection to have a defined schema before Unstructured can write to the collection. The minimum viable
schema for Unstructured contains only the fields `element_id`, `embeddings`, `record_id`, and `text`, as follows.
`type` is an optional field, but highly recommended.

This example code demonstrates the use of the
[Python SDK for Milvus](https://pypi.org/project/pymilvus/) to create a collection with this schema,
targeting Milvus on IBM watsonx.data. For the `MilvusClient` arguments to connect to other types of Milvus deployments, see your Milvus provider's documentation:

```python Python theme={null}
import os

from pymilvus import (
    MilvusClient,
    FieldSchema,
    DataType,
    CollectionSchema
)

DATABASE_NAME   = "default"
COLLECTION_NAME = "my_collection"

client = MilvusClient(
    uri="https://" +
        os.getenv("MILVUS_USER") + 
        ":" + 
        os.getenv("MILVUS_PASSWORD") + 
        "@" + 
        os.getenv("MILVUS_GRPC_HOST") + 
        ":" + 
        os.getenv("MILVUS_GRPC_PORT"),
    db_name=DATABASE_NAME
)

# IMPORTANT: The number of dimensions for the "embeddings" field that
# follows must match the number of dimensions for the embedding model 
# that you plan to use.
fields = [
    FieldSchema(name="element_id", dtype=DataType.VARCHAR, is_primary=True, max_length=200),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=384),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=200),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="type", dtype=DataType.VARCHAR,max_length=200, nullable=True) # Optional, but highly recommended.
]

schema = CollectionSchema(fields=fields)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema,
    using=DATABASE_NAME
)

index_params = client.prepare_index_params()

index_params.add_index(
    field_name="embeddings",
    metric_type="COSINE",
    index_type="IVF_FLAT",
    params={"nlist": 1024}
)

client.create_index(
    collection_name=COLLECTION_NAME,
    index_params=index_params
)

client.load_collection(collection_name=COLLECTION_NAME)
```

For objects in the `metadata` field that Unstructured produces and that you want to store in Milvus, you must create fields in your Milvus collection schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You could create corresponding fields in your Milvus collection schema by using the following field names, data types, and related settings for the
Python SDK for Milvus:

```python theme={null}
# ...
fields = [
    # Define minimum viable required fields: "element_id", "embeddings", "record_id", and "text".
    # "type" is an optional field, but highly recommended.
    FieldSchema(name="element_id", dtype=DataType.VARCHAR, is_primary=True, max_length=200),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=1536),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=200),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="type",dtype=DataType.VARCHAR,max_length=200, nullable=True),
    # Define optional fields, such as these ones related to the "metadata" field.
    FieldSchema(name="is_extracted", dtype=DataType.VARCHAR, max_length=5, nullable=True),
    FieldSchema(name="coordinates_points", dtype=DataType.JSON, nullable=True),
    FieldSchema(name="coordinates_system", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="coordinates_layout_width", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="coordinates_layout_height", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="filetype", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="languages", dtype=DataType.ARRAY, element_type=DataType.VARCHAR, max_length=64, max_capacity=10, nullable=True),
    FieldSchema(name="page_number", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="image_mime_type", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="filename", dtype=DataType.VARCHAR, max_length=256, nullable=True),
    FieldSchema(name="data_source_url", dtype=DataType.VARCHAR, max_length=1024, nullable=True),
    FieldSchema(name="data_source_record_locator_protocol", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="data_source_record_locator_remote_file_path", dtype=DataType.VARCHAR, max_length=1024, nullable=True),
    FieldSchema(name="entities_items", dtype=DataType.JSON, nullable=True),
    FieldSchema(name="entities_relationships", dtype=DataType.JSON, nullable=True)
]
# ...
```

<Note>
  Fields with a `DataType.VARCHAR` data type are limited to a maximum length of 65,535 characters. Attempting to exceed this character count
  will cause Unstructured to throw errors when attempting to write to a Milvus collection, and the associated Unstructured job could fail.
  For example, `metadata` fields that typically exceed these character counts include `image_base64` and `orig_elements`.
</Note>

The Milvus connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[milvus]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `MILVUS_URI` - The Milvus instance's URI, represented by `--uri` (CLI) or `uri` (Python).
* `MILVUS_USER` and `MILVUS_PASSWORD`, or `MILVUS_TOKEN` - The username and password, or token, to access the instance. This is represented by `--user` and `--password`, or `--token` (CLI); or `user` and `password`, or `token` (Python).
* `MILVUS_DB` - The database's name, represented by `--db-name` (CLI) or `db_name` (Python).
* `MILVUS_COLLECTION` - The collection's name, represented by `--collection-name` (CLI) or `collection_name` (Python).
* `MILVUS_FIELDS_TO_INCLUDE` - A list of fields to include a comma-separated list (CLI) or an array of strings (Python), represented by `--field-to-include` (CLI) or `fields_to_include` (Python).

Additional settings include:

* To emit the `metadata` field's child fields directly into the output, include `--flatten-metadata` (CLI) or `flatten_metadata=True` (Python). This is the default if not specified.
* To keep the `metadata` field with its child fields intact in the output, include `--no-flatten-metadata` (CLI) or `flatten_metadata=False` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    milvus \
      --uri $MILVUS_URI \
      --user $MILVUS_USER \
      --password $MILVUS_PASSWORD \
      --db-name $MILVUS_DB \
      --collection-name $MILVUS_COLLECTION \
      --fields-to-include type,element_id,text,embeddings
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.milvus import (
      MilvusConnectionConfig,
      MilvusAccessConfig,
      MilvusUploadStagerConfig,
      MilvusUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MilvusConnectionConfig(
              access_config=MilvusAccessConfig(
                  password=os.getenv("MILVUS_PASSWORD")
              ),
              uri=os.getenv("MILVUS_URI"),
              user=os.getenv("MILVUS_USER"),
              db_name=os.getenv("MILVUS_DB")
          ),
          stager_config=MilvusUploadStagerConfig(fields_to_include=["type", "element_id", "text", "embeddings"]),
          uploader_config=MilvusUploaderConfig(collection_name=os.getenv("MILVUS_COLLECTION"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# MongoDB
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/mongodb



Batch process all your records to store structured outputs in MongoDB.

The requirements are as follows.

<iframe title="YouTube video player" />

The MongoDB requirements for a MongoDB Atlas deployment include:

<Warning>
  For MongoDB Atlas, SCRAM-SHA-1 is not supported for authentication. This means that cluster types that only
  include SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, and **Serverless**, are **not** supported.
  Unstructured only supports SCRAM-SHA-256 for MongoDB Atlas, which is cryptographically stronger than SCRAM-SHA-1.

  If you try to test or use a connector that refers to a cluster type that only includes SCRAM-SHA-1, the
  operation will fail, and you will get an error message similar to the following:
  `[digital envelope routines] unsupported`.
</Warning>

* A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).

* A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). Be sure to **not**
  select a cluster type that only includes SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, or **Serverless**.

* The cluster must be reachable from your application environment, for example by adding IP addresses to your IP access list. [Learn more](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).

* The cluster must be configured to allow IP address. [Learn how](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries).

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

* The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).

* The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).

* The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).

  <Note>
    For the destination connector, Unstructured recommends that all documents in the target collection have a field
    named `record_id` with a `String` data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the collection or, in some cases, the operation could fail altogether.
  </Note>

* The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

  ```text theme={null}
  mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
  ```

  To get the connection string in MongoDB Atlas, do the following:

  1. Log in to your MongoDB Atlas console.
  2. In the sidebar, under **Databases**, click **Clusters**.
  3. Click on the cluster you want to connect to.
  4. Click **Connect**.
  5. Click **Drivers**.
  6. Under **Add your connection string into your application code**, copy the connection string.
     You can then close the **Connect** dialog in MongoDB Atlas.

     Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

  [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

The MongoDB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[mongodb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

For a MongoDB Atlas deployment, the following environment variables:

* `MONGODB_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

* `MONGODB_COLLECTION` - The name of the collection in the database, represented by `--collection` (CLI) or `collection` (Python).

* `MONGODB_URI` - The connection string for the cluster, represented by `--uri` (CLI) or `uri` (Python).

For a local MongoDB server, the following environment variables:

* `MONGODB_HOST` - The host for the local MongoDB server, represented by `--host` (CLI) or `host` (Python).

* `MONGODB_PORT` - The port for the local MongoDB server, represented by `--port` (CLI) or `port` (Python).

Now call the Unstructured Ingest CLI or Unstructured Ingest Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    mongodb \
      --uri $MONGODB_URI \
      --database $MONGODB_DATABASE \
      --collection $MONGODB_COLLECTION
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.mongodb import (
      MongoDBAccessConfig,
      MongoDBConnectionConfig,
      MongoDBUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MongoDBConnectionConfig(
              access_config=MongoDBAccessConfig(
                  uri=os.getenv("MONGODB_URI")
              ),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION")
          ),
          uploader_config=MongoDBUploaderConfig(batch_size=100)
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# MotherDuck
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/motherduck



Batch process all your records to store structured outputs in a MotherDuck account.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [MotherDuck account](https://app.motherduck.com).

* A [MotherDuck access token](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck/#creating-an-access-token) for the account.

* A database in the account.

  * [Create a database](https://motherduck.com/docs/sql-reference/motherduck-sql-reference/create-database/).
  * [List available databases](https://motherduck.com/docs/key-tasks/database-operations/basics-operations/#listing-databases).

  You can run commands to manage MotherDuck databases, schemas, tables, and more in the
  [MotherDuck UI](https://motherduck.com/docs/getting-started/motherduck-quick-tour/) or for example by connecting to MotherDuck with the
  [DuckDB CLI](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/connecting-to-motherduck/).

* A schema in the target database.

  * [Create a schema](https://duckdb.org/docs/sql/statements/create_schema.html).
  * You can list available schemas and their parent catalogs by running the following command in the MotherDuck UI or the DuckDB CLI:

    ```sql theme={null}
    SELECT * FROM information_schema.schemata;
    ```

  The MotherDuck connector uses the default schema name of `main` if not otherwise specified.

* A table in the target schema.

  * [Create a table](https://duckdb.org/docs/sql/statements/create_table).
  * You can list available tables in a schema by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog and schema names:

    ```sql theme={null}
    USE <catalog_name>.<schema_name>;
    SHOW TABLES;
    ```

  The MotherDuck connector uses the default table name of `elements` if not otherwise specified.

  For maximum compatibility, Unstructured recommends the following table schema:

  ```sql theme={null}
  CREATE TABLE elements (
      id VARCHAR,
      element_id VARCHAR,
      text TEXT,
      embeddings FLOAT[],
      type VARCHAR,
      system VARCHAR,
      layout_width DECIMAL,
      layout_height DECIMAL,
      points TEXT,
      url TEXT,
      version VARCHAR,
      date_created INTEGER,
      date_modified INTEGER,
      date_processed DOUBLE,
      permissions_data TEXT,
      record_locator TEXT,
      category_depth INTEGER,
      parent_id VARCHAR,
      attached_filename VARCHAR,
      filetype VARCHAR,
      last_modified TIMESTAMP,
      file_directory VARCHAR,
      filename VARCHAR,
      languages VARCHAR[],
      page_number VARCHAR,
      links TEXT,
      page_name VARCHAR,
      link_urls VARCHAR[],
      link_texts VARCHAR[],
      sent_from VARCHAR[],
      sent_to VARCHAR[],
      subject VARCHAR,
      section VARCHAR,
      header_footer_type VARCHAR,
      emphasized_text_contents VARCHAR[],
      emphasized_text_tags VARCHAR[],
      text_as_html TEXT,
      regex_metadata TEXT,
      detection_class_prob DECIMAL,
      partitioner_type VARCHAR
  );
  ```

  You can list the schema of a table by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog, schema, and table names:

  ```sql theme={null}
  USE <catalog_name>.<schema_name>;
  DESCRIBE TABLE <table_name>;
  ```

The MotherDuck connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[duckdb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `MOTHERDUCK_MD_TOKEN` - The access token for the target MotherDuck account, represented by `--md-token` (CLI) or `md_token` (Python).
* `MOTHERDUCK_DATABASE` - The name of the target database in the account, represented by `--database` (CLI) or `database` (Python).
* `MOTHERDUCK_DB_SCHEMA` - The name of the target schema in the database, represented by `--db-schema` (CLI) or `db_schema` (Python).
* `MOTHERDUCK_TABLE` - The name of the target table in the schema, represented by `--table` (CLI) or `table` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    motherduck \
      --md-token $MOTHERDUCK_MD_TOKEN \
      --database $MOTHERDUCK_DATABASE \
      --db-schema $MOTHERDUCK_DB_SCHEMA \
      --table $MOTHERDUCK_TABLE
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.duckdb.motherduck import (
      MotherDuckAccessConfig,
      MotherDuckConnectionConfig,
      MotherDuckUploadStagerConfig,
      MotherDuckUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=MotherDuckConnectionConfig(
              access_config=MotherDuckAccessConfig(md_token=os.getenv("MOTHERDUCK_MD_TOKEN")),
              database=os.getenv("MOTHERDUCK_DATABASE"),
              db_schema=os.getenv("MOTHERDUCK_DB_SCHEMA"),
              table=os.getenv("MOTHERDUCK_TABLE")
          ),
          stager_config=MotherDuckUploadStagerConfig(),
          uploader_config=MotherDuckUploaderConfig(batch_size=50)
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Neo4j
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/neo4j



Batch process all your records to store structured outputs in a Neo4j account.

The requirements are as follows.

* A [Neo4j deployment](https://neo4j.com/deployment-center/).

  * For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local Neo4j deployments are not supported.
  * For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local Neo4j deployments are supported.

  The following video shows how to set up a Neo4j Aura deployment:

  <iframe title="YouTube video player" />

* The username and password for the user who has access to the Neo4j deployment. The default user is typically `neo4j`.

  * For a Neo4j Aura instance, the defaut user's is typically set when the instance is created.
  * For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, the default user is typically set during the deployment process.
  * For a local Neo4j deployment, you can [set the default user's initial password](https://neo4j.com/docs/operations-manual/current/configuration/set-initial-password/) or [recover an admin user and its password](https://neo4j.com/docs/operations-manual/current/authentication-authorization/password-and-user-recovery/).

* The connection URI for the Neo4j deployment, which starts with `neo4j://`, `neo4j+s://`, `bolt://`, or `bolt+s://`; followed by `localhost` or the host name; and sometimes ending with a colon and the port number (such as `:7687`). For example:

  * For a Neo4j Aura deployment, browse to the target Neo4j instance in the Neo4j Aura account and click **Connect > Drivers** to get the connection URI, which follows the format `neo4j+s://<host-name>`. A port number is not used or needed.
  * For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, see
    [Neo4j on AWS](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-aws/),
    [Neo4j on Azure](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-azure/), or
    [Neo4j on GCP](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-gcp/)
    for details about how to get the connection URI.
  * For a local Neo4j deployment, the URI is typically `bolt://localhost:7687`
  * For other Neo4j deployment types, see the deployment provider's documentation.

  [Learn more](https://neo4j.com/docs/browser-manual/current/operations/dbms-connection).

* The name of the target database in the Neo4j deployment. A default Neo4j deployment typically contains two standard databases: one named `neo4j` for user data and another
  named `system` for system data and metadata. Some Neo4j deployment types support more than these two databases per deployment;
  Neo4j Aura instances do not.

  * [Create additional databases](https://neo4j.com/docs/operations-manual/current/database-administration/standard-databases/create-databases/)
    for a local Neo4j deployment that uses Enterprise Edition; or for Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP deployments.
  * [Get a list of additional available databases](https://neo4j.com/docs/operations-manual/current/database-administration/standard-databases/listing-databases/)
    for a local Neo4j deployment that uses Enterprise Edition; or for Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP deployments.

The Neo4j connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[neo4j]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `NEO4J_USERNAME` - The name of the target user with access to the target Neo4j deployment, represented by `--username` (CLI) or `username` (Python).
* `NEO4J_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
* `NEO4J_URI` - The connection URI for the deployment, represented by `--uri` (CLI) or `uri` (Python).
* `NEO4J_DATABASE` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    neo4j \
      --username $NEO4J_USERNAME \
      --password $NEO4J_PASSWORD \
      --uri $NEO4J_URI \ # <scheme>://<host>:<port>
      --database $NEO4J_DATABASE \
      --batch-size 100
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.neo4j import (
      Neo4jAccessConfig,
      Neo4jConnectionConfig,
      Neo4jUploadStagerConfig,
      Neo4jUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=Neo4jConnectionConfig(
              access_config=Neo4jAccessConfig(password=os.getenv("NEO4J_PASSWORD")),
              username=os.getenv("NEO4J_USERNAME"),
              uri=os.getenv("NEO4J_URI"),
              database=os.getenv("NEO4J_DATABASE"),
          ),
          stager_config=Neo4jUploadStagerConfig(),
          uploader_config=Neo4jUploaderConfig(batch_size=100)
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>

## Graph Output

The graph output of the Neo4j destination connector is represented in the following diagram:

```mermaid theme={null}
graph BT
    subgraph dn [Document Node]
    D[Document]
    end
    style dn stroke-dasharray: 5
    
    subgraph en [Element Nodes]
    UE1[UnstructuredElement]
    UE2[UnstructuredElement]
    UE3[UnstructuredElement]
    UE4[UnstructuredElement]
    UE5[UnstructuredElement]
    UE6[UnstructuredElement]
    end
    style en stroke-dasharray: 5
    
    UE1 -->|PART_OF_DOCUMENT| D
    UE2 -->|PART_OF_DOCUMENT| D
    UE3 -->|PART_OF_DOCUMENT| D
    UE4 -->|PART_OF_DOCUMENT| D
    UE5 -->|PART_OF_DOCUMENT| D
    UE6 -->|PART_OF_DOCUMENT| D

    subgraph cn [Chunk Nodes]
    C1[Chunk]
    C2[Chunk]
    C3[Chunk]
    C4[Chunk]
    end
    style cn stroke-dasharray: 5
    
    C1 -->|NEXT_CHUNK| C2
    C2 -->|NEXT_CHUNK| C3
    C3 -->|NEXT_CHUNK| C4

    C1 -->|PART_OF_DOCUMENT| D
    C2 -->|PART_OF_DOCUMENT| D
    C3 -->|PART_OF_DOCUMENT| D
    C4 -->|PART_OF_DOCUMENT| D

    UE1 -.->|PART_OF_CHUNK| C1
    UE2 -.->|PART_OF_CHUNK| C1
    UE3 -.->|PART_OF_CHUNK| C2
    UE4 -.->|PART_OF_CHUNK| C3
    UE5 -.->|PART_OF_CHUNK| C4
    UE6 -.->|PART_OF_CHUNK| C4
```

[View the preceding diagram in full-screen mode](https://mermaid.live/view#pako:eNqFlN9vgjAQx_-Vps-6REEfeFiyFZYli7hskCyTxXS0ihFaU9oHo_7vq_IjgIzyxN330157d70TjDmh0IFbgQ8JeA4iBvSXq9_CQRhYuTxWGWUS-Br9KQC39pYOyki5VB5Tel2XS8H3dExwnmAh8NEBs4LohKA6hJfSOkJe7hh6k1XI9C4qlkpQUjK1Oh1UrUHVHlRng-p8QO1kgRqzoC8JxuPH8_vTR7BevqzdJQoXnh-cgVvf0wRYJsA2ATMTMP8f6FQz1tVEiWL7Vi3RpHBW5rRtWm3TbpmdnMbGnKIipb73FazRa-i_nXXAKvC9ZFWHuJfs6nrIUCVkKBIy1AjZpgTfGuWhwVRnnDT6ZFC3-vVpo0v6dKvRJH263eiRXh2OYEZFhndEj5nTlY6gTPSriaCjfwndYJXKCEbsolGsJP88shg6-onRERRcbRPobHCaa0sdCJbU3WHdbFmFHDD75jyrIUp2kotFMddu4-3yB3k-fcg).

In the preceding diagram:

* The `Document` node represents the source file.
* The `UnstructuredElement` nodes represent the source file's Unstructured `Element` objects, before chunking.
* The `Chunk` nodes represent the source file's Unstructured `Element` objects, after chunking.
* Each `UnstructuredElement` node has a `PART_OF_DOCUMENT` relationship with the `Document` node.
* Each `Chunk` node also has a `PART_OF_DOCUMENT` relationship with the `Document` node.
* Each `UnstructuredElement` node has a `PART_OF_CHUNK` relationship with a `Chunk` element.
* Each `Chunk` node, except for the "last" `Chunk` node, has a `NEXT_CHUNK` relationship with its "next" `Chunk` node.

Learn more about [document elements](/concepts/document-elements) and [chunking](/concepts/chunking).

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

```text theme={null}
MATCH path=(source)-[relationship]->(target)
RETURN path
```

Query for `Chunk` to `Document` relationships:

```text theme={null}
MATCH (chunk:Chunk)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN chunk, relationship, doc
```

Query for `UnstructuredElement` to `Document` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, relationship, doc
```

Query for `UnstructuredElement` to `Chunk` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
RETURN element, relationship, chunk
```

Query for `Chunk` to `Chunk` relationships:

```text theme={null}
MATCH (this:Chunk)-[relationship:NEXT_CHUNK]->(previous:Chunk)
RETURN this, relationship, previous
```

Query for `UnstructuredElement` to `Chunk` to `Document` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[ecrelationship:PART_OF_CHUNK]-(chunk:Chunk)-[cdrelationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, ecrelationship, chunk, cdrelationship, doc
```

Query for `UnstructuredElements` containing the text `jury`, and show their `Chunk` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE element.text =~ '(?i).*jury.*'
RETURN element, relationship, chunk
```

Query for the `Chunk` with the specified `id`, and show its `UnstructuredElement` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE chunk.id = '731508bf53637ce4431fe93f6028ebdf'
RETURN element, relationship, chunk
```

Additionally, for the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API's workflow operations](/api-reference/workflow/),
when a [Named entity recognition (NER)](/concepts/enriching/ner) DAG node is added to a custom workflow,
any recognized entities are output as `Entity` nodes in the graph.

This additional graph output of the Neo4j destination connector is represented in the following diagram:

```mermaid theme={null}
graph TD
    Chunk -->|HAS_ENTITY| Entity
    Entity -->|ENTITY_TYPE| Entity
```

In the preceding diagram:

* The `Chunk` node represents one of the source file's Unstructured `Element` objects, after chunking.
* The `Entity` node represents a recognized entity.
* A `Chunk` node can have `HAS_ENTITY` relationships with `Entity` nodes.
* An `Entity` node can have `ENTITY_TYPE` relationships with other `Entity` nodes.

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

```text theme={null}
MATCH path=(source)-[relationship]->(target)
RETURN path
```

Query for `Entity` to `Entity` relationships:

```text theme={null}
MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
RETURN child, relationship, parent
```

Query for `Entity` nodes containing the text `PERSON`, and show their `Entity` relationships:

```text theme={null}
MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id = 'PERSON'
RETURN child, relationship, parent
```

Query for `Entity` nodes containing the text `amendment`, and show their `Chunk` relationships:

```text theme={null}
MATCH (element:Chunk)-[relationship:HAS_ENTITY]->(entity:Entity)
WHERE entity.id =~ '(?i).*amendment.*'
RETURN element, relationship, entity
```

QUERY FOR `Entity` nodes containing the text `PERSON`, and show their `Entity` to `Entity` to `Chunk` relationships:

```text theme={null}
MATCH (chunk:Chunk)-[ccrelationship:HAS_ENTITY]-(child:Entity)-[cprelationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id =~ 'PERSON'
RETURN chunk, ccrelationship, child, cprelationship, parent
```


# OneDrive
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/onedrive



Batch process all your records to store structured outputs in a OneDrive account.

The requirements are as follows.

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* The OneDrive and SharePoint Online plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.

  The following video shows how to get a path:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials, and a username and password. Both authentication types require a Microsoft Entra ID app registration.

  <Note>
    The OneDrive connector does not support any other authentication methods, such as multifactor (MFA) or passwordless authentication.
  </Note>

  You will need to provide the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration that has access to the target OneDrive account, and
  the app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Files.ReadWrite.All` (if both reading and writing are needed)
  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
  * `Directory.Read.All`

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* Both authentication types also require the User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

* For username and password authentication, you will also need the password for the target UPN.

The OneDrive connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[onedrive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `ONEDRIVE_PATH` - The path to the target folder in the OneDrive account, starting with the account's root folder (for example `my-folder/my-subfolder`), represented by `--path` (CLI) or `path` (Python).
* `ONEDRIVE_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account, represented by `--client-id` (CLI) or `client_id` (Python).
* `ONEDRIVE_CLIENT_CRED` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
* `ONEDRIVE_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).
* `ONEDRIVE_USER_PNAME` - The User Principal Name (UPN) for the OneDrive user account in Entra ID (typically the user's email address), represented by `--user-pname` (CLI) or `user_pname` (Python).
* `ONEDRIVE_USER_PASSWORD` - The password for the target UPN, represented by `--password` (CLI) or `password` (Python).
* `ONEDRIVE_AUTHORITY_URL` - The authentication token provider URL for the Entra ID app registration (the default is `https://login.microsoftonline.com` if not otherwise specified), represented by `--authority-url` (CLI) or `authority_url` (Python).

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    onedrive \
      --client-cred $ONEDRIVE_CLIENT_CRED \
      --client-id $ONEDRIVE_CLIENT_ID \
      --user-pname $ONEDRIVE_USER_PNAME \
      --password $ONEDRIVE_USER_PASSWORD \ # For username and password authentication.
      --tenant $ONEDRIVE_TENANT \
      --authority-url $ONEDRIVE_AUTHORITY_URL \
      --remote-url $ONEDRIVE_PATH \
      --prefix "onedrive://"
  ```

  ```bash Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.onedrive import (
      OnedriveConnectionConfig,
      OnedriveAccessConfig,
      OnedriveUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=OnedriveConnectionConfig(
              access_config=OnedriveAccessConfig(
                  client_cred=os.getenv("ONEDRIVE_CLIENT_CRED"),
                  password=os.getenv("ONEDRIVE_USER_PASSWORD") # For username and password authentication.
              ),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL")
          ),
          uploader_config=OnedriveUploaderConfig(
              remote_url=os.getenv("ONEDRIVE_PATH"),
              prefix="onedrive://"
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# OpenSearch
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/opensearch



Batch process all your records to store structured outputs in OpenSearch.

The requirements are as follows.

## Set up OpenSearch

Supported OpenSearch installations vary by product:

* **[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):** Non-local OpenSearch only. Local OpenSearch instances are not supported.
* **[Unstructured Ingest](/open-source/ingestion/overview):** Both local and non-local OpenSearch instances are supported.

### Set up an AWS OpenSearch Service domain

To set up an [AWS OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) domain, complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.

4. Click **Create domain**.

5. In the **Name** tile, for **Domain name**, enter some unique domain name for your new OpenSearch domain.

6. In the **Domain creation method** tile, select **Easy create**. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With **Standard create**, you must enable FGAC manually.) [Standard create method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html).

7. In the **Engine options** tile, for **Version**, AWS recommends that you select the latest version.

8. In the **Network** tile, for **Network**, select a network access method.
   For faster setup, this example uses the **Public access** method.
   [VPC access method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#prerequisites-vpc-endpoints).

9. For **IP address type**, select **Dual-stack mode**.

10. In the **Fine-grained access control** (FGAC) tile, do one of the following:

    * If you want to use an existing AWS IAM user in the AWS account as the domain's master user, then for **Master user**, select **Set IAM ARN as master user**. Then enter the IAM ARN for the master user in the **IAM ARN** box.
    * If you want to create a master user and password as the domain's master user instead, then for **Master user**, select **Create master user**. Then specify some username and password for this
      new master user by filling in the **Master username**, **Master password**, and **Confirm master password** fields. Make
      sure to save the master user's password in a secure location.

11. Click **Create**.

12. After the domain is created, you must allow Unstructured to access the domain, as follows:

    a. If the new domain's settings page is not already showing, open it as follows:
    in your Amazon OpenSearch Service console, on the sidebar, expand **Managed clusters**, and then click **Domains**. Then,
    in the list of available domains, click the name of the newly created domain.<br />
    b. On the **Security configuration** tab, click **Edit**.<br />
    c. In the **Access policy** tile, for **Domain access policy**, select **Only use fine-grained access control**.<br />
    d. Click **Clear policy**. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).<br />
    e. Click **Save changes**.

### Set up an Amazon OpenSearch Serverless collection

To set up an [Amazon OpenSearch Serverless collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-create-console.html), complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Serverless**, and then click **Dashboard**.

4. Click **Create collection**.

5. In the **Collection details** tile, for **Collection name**, enter some unique name for your new OpenSearch Serverless collection.
   Optionally, for **Description**, enter some meaningful description for your new collection.

6. For **Collection type**, select **Search**.

   <Note>
     Unstructured does not support the **Vector search** collection type. If you need vector search support, you can either continue
     with these steps to use the **Search** collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead.
     However, note that the Amazon OpenSearch Serverless **Search** collection type is not as optimal as the **Vector search** collection type.
   </Note>

7. In the **Collection creation method** tile, select **Standard create**.

8. For **Encryption**, choose an AWS KMS key type.

9. For **Network access settings**, choose an **Access type**.

10. For **Resource type**, select both **Enable access to OpenSearch endpoint** and **Enable access to OpenSearch Dashboards**.

11. Click **Next**.

12. In the **Definition method** tile, select **JSON**.

13. In the **JSON editor** box, enter the following JSON, replacing the following placeholders:

    * Replace `<collection-name>` with the name of the new OpenSearch Serverless collection.
    * Replace `<account-id>` with the target AWS account ID.
    * Replace `<user-id>` with the ID of the target AWS IAM user.

    ```json theme={null}
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    ```

14. Click **Next**.

15. For **Data access policy settings**, select **Create as a new data access policy**.

16. In the **Name and description** tile, enter some unique name and an optional description for the new data access policy.

17. Click **Next**.

18. Enter any desired index details, and click **Next** again. For example:

    a. For **Index name**, enter the name of the new index in the collection.<br />
    b. For **Automatic Semantic Enrichment fields**, click **Add**, enter `embeddings` for **Automatic Semantic Enrichment field name**, click **Add**, and click **Confirm**.<br />
    c. For **Lexical search fields**, click **Add**, enter `text` for **Field name** and select **Text** for **Data type**, click **Add**, and click **Confirm**.<br />

19. Click **Submit**.

### Set up a local OpenSearch instance

The following video shows how to set up a [local OpenSearch](https://opensearch.org/downloads.html) instance.

<iframe title="YouTube video player" />

### Get the host URL

Find your host URL in the AWS console using the steps for your OpenSearch type.

#### Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.
4. In the list of available domains, click the name of your domain.
5. In the **General information** tile, copy the value of **Domain endpoint v2 (dual stack)**.

#### Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Serverless**, and then click **Dashboard**.
4. In the list of available collections, click the name of your collection.
5. On the **Overview** tab, in the **Endpoint** tile, copy the value of **OpenSearch endpoint**.

#### Local OpenSearch instance

Your local instance URL depends on how OpenSearch is installed and configured. For guidance, see [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) in the OpenSearch documentation.

### Create a search index

The name of the search index on the instance is required.

For the destination connector, if you need to create an index and you're using a master user and password as the domain's master user, you can use for example the following `curl` command. Replace the following placeholders:

* Replace `<host>` with the instance's host URL.
* Replace `<port>` with the instance's port number, which is typically `443` (for encrypted connections, and less commonly `9200` for unencrypted connections).
* Replace `<master-username>` with the master user's name, and replace `<master-password>` with the master user's password.
* Replace `<index-name>` with the name of the new search index on the instance.
* Replace `<index-schema>` with the schema for the new search index on the instance. A schema is optional; see the explanation
  following this `curl` command for more information.

```bash theme={null}
curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
```

If you're using an existing AWS IAM user as the domain's master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the preceding `curl` command. To learn how, see [create-index](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-index.html) in the AWS CLI Command Reference.

For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

For objects in the `metadata` field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index's schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
required by Unstructured whenever you create your own index schema:

* `element_id`
* `record_id`, which is required by Unstructured for intelligent record updates.
* `type`, which is not required, but highly recommended.
* `text`
* `embeddings` if embeddings are generated; make sure to set `dimension` to the same number of dimensions as the embedding model generates.

```json theme={null}
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

See also:

* [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/)
* [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
* [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
* [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
* [Unstructured document elements and metadata](/concepts/document-elements)

### Set up master user authentication

<Note>
  If you are using Enterprise Connect on a [dedicated instance](/business/dedicated-instances/overview), you do not need **master user credentials**. Skip to the **Enterprise Connect** section on this page.
</Note>

For non-local OpenSearch instances, or if you're using basic authentication to a local OpenSearch instance, you will need the master user's name and password.

For local OpenSearch instances, if you're using certificates for authentication instead of basic authentication, you will need:

* The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
* The path to the combined private key and certificate file, or
* The paths to the separate private key and certificate files.

To learn more, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

The OpenSearch connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[opensearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `OPENSEARCH_HOST_URL` - The OpenSearch instance's host URL, which typically takes the form of `https://<host>:<port>`, and is represented by `--hosts` (CLI) or `hosts` (Python).
* `OPENSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using basic authentication to the instance:

* `OPENSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
* `OPENSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using certificates for authentication instead:

* `OPENSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by `--ca-certs` (CLI) or `ca_certs` (Python).
* `OPENSEARCH_CLIENT_CERT` - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by `--client-cert` (CLI) or `client_cert` (Python).
* `OPENSEARCH_CLIENT_KEY` - The path to the private key file, if `OPENSEARCH_CLIENT_CERT` refers to just the certificate file. This is represented by `--client-key` (CLI) or `client_key` (Python).

Additional related settings include:

* `--use-ssl` (CLI) or `use_ssl=True` (Python) to use SSL for the connection.
* `--verify-certs` (CLI) or `verify_certs=True` (Python) to verify SSL certificates.
* `--ssl-show-warn` (CLI) or `ssl_show_warn=True` (Python) to show a warning when verifying SSL certificates is disabled.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    opensearch \
      --hosts $OPENSEARCH_HOST_URL \
      --username $OPENSEARCH_USERNAME \
      --password $OPENSEARCH_PASSWORD \
      --index-name $OPENSEARCH_INDEX_NAME
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.elasticsearch.opensearch import (
      OpenSearchConnectionConfig,
      OpenSearchAccessConfig,
      OpensearchUploadStagerConfig,
      OpensearchUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=OpenSearchConnectionConfig(
              access_config=OpenSearchAccessConfig(
                  password=os.getenv("OPENSEARCH_PASSWORD"),
                  use_ssl=True
              ),
              hosts=[os.getenv("OPENSEARCH_HOST_URL")],
              username=os.getenv("OPENSEARCH_USERNAME")
          ),
          stager_config=OpensearchUploadStagerConfig(
              index_name=os.getenv("OPENSEARCH_INDEX_NAME")
          ),
          uploader_config=OpensearchUploaderConfig(
              index_name=os.getenv("OPENSEARCH_INDEX_NAME")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Overview
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/overview



Ingest and process data from a storage with a source connector, and move the results to the data storage platform of
your choice using a destination connector.

All the examples for destination connector use local source connector. You can replace it with any other source connector.

We are constantly adding new destination connectors and if you don't see your favorite platform let us know in our community Slack.


# Pinecone
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/pinecone



<NewDocument />

Batch process all your records to store structured outputs in Pinecone.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Pinecone account. [Get an account](https://app.pinecone.io/).

* A Pinecone API key in your Pinecone account. To create an API key, do the following:

  1. [Sign in to your Pinecone account and open the API keys page](https://app.pinecone.io/organizations/-/projects/-/keys).
  2. Click **+ API key**.
  3. For **API key name**, enter some descriptive name for the API key.
  4. Click **Create key**.
  5. Copy the generated API key to some secure location. You will not be able to access this API key again after you close the dialog.

* A Pinecone serverless index in your Pinecone account.

  Creating a serverless index on your own is optional.
  An index is not required to exist in advance.

  When you set up the connector, at runtime, the index behavior is as follows:

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * Your workflow must contain an embedder node, and the embedder node must specify the embedding model that Unstructured will use to generate the embeedings.
  * If an existing index name is specified,
    and the number of dimensions that Unstructured generates does not match the number of dimensions that are specified in the existing index's embedding settings, the run will fail.
    You must change the number of dimensions in your workflow's embedder node or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The
    new index's name will be `u<short-workflow-id>-<short-embedding-model-name>-<number-of-dimensions>`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * If an existing index name is specified, and Unstructured generates embeddings,
    but the number of dimensions that are generated does not match the existing index's embedding settings, the run will fail.
    You must change your Unstructured embedding settings or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The new index's name will be `unstructuredautocreated`.

  <Note>
    If you create a new index or use an existing one, Unstructured recommends that all records in the target index have a field
    named `record_id` with a string data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the index or, in some cases, the operation could fail altogether.
  </Note>

  To create a serverless index on your own, do the following:

  1. [Sign in to your Pinecone account and open the Create a new index page](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless).

  2. For **Enter index name**, enter some descriptive name for the index.

  3. For **Configuration**, select the check box labelled **Custom settings**, or click the tile labelled **Manual configuration**.

     <Warning>
       Do not click any of the other tiles, such as **text-embedding-3-large**. Clicking any of these other tiles will cause Pinecone to generate embeddings instead of
       having Unstructured generate them. If Pinecone generates embeddings instead of Unstructured, this could cause any related Unstructured workflows to fail.
     </Warning>

  4. For **Vector type**, select **Dense**.

  5. For **Dimension**, enter the number of dimensions for the embeddings that Unstructured will generate.

     <Warning>
       The number of dimensions that you enter here must match the number of dimensions for the embedding model that you use in any related Unstructured workflows or ingestion pipelines. If these numbers do not
       match in both places, this could cause any related Unstructured workflows or ingestion pipelines to fail.
     </Warning>

  6. For **Metric**, select **cosine**.

  7. Leave **Capacity mode** set to **Serverless**.

  8. You can leave **Cloud provider** and **Region** set to their default values, or you can select a cloud provider and region that is closest to you, if available.

  9. Click **Create index**.

* Within a Pinecone serverless index, custom [namespaces](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces) are supported but are not required.

The Pinecone connector dependencies.

```bash theme={null}
pip install "unstructured-ingest[pinecone]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `PINECONE_API_KEY` - The Pinecone API, represented by `--api-key` (CLI) or `api_key` (Python, in the `PineconeAccessConfig` object).
* `PINECONE_INDEX_NAME` - The Pinecone serverless index name, represented by `--index-name` (CLI) or `index_name` (Python). If no value is provided, see the beginning of this article for the behavior at run time.
* `PINECONE_NAMESPACE_NAME` - The name of any custom namespace in the Pinecone index to upsert data into, represented by `--namespace` (CLI) or `namespace` (Python). If no value is provided, the default namespace of `default` will be used.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

<iframe title="YouTube video player" />

This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --strategy hi_res \
      --chunk-elements \
      --embedding-provider huggingface \
      --num-processes 2 \
      --verbose \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    pinecone \
      --api-key "$PINECONE_API_KEY" \
      --index-name "$PINECONE_INDEX_NAME" \
      --namespace "$PINECONE_NAMESPACE_NAME" \
      --batch-size 80
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.pinecone import (
      PineconeConnectionConfig,
      PineconeAccessConfig,
      PineconeUploaderConfig,
      PineconeUploadStagerConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=PineconeConnectionConfig(
              access_config=PineconeAccessConfig(
                  api_key=os.getenv("PINECONE_API_KEY")
              ),
              index_name=os.getenv("PINECONE_INDEX_NAME")
          ),
          stager_config=PineconeUploadStagerConfig(),
          uploader_config=PineconeUploaderConfig(
              batch_size=100,
              namespace=os.getenv("PINECONE_NAMESPACE_NAME")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# PostgreSQL
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/postgresql



Batch process all your records to store structured outputs in a PostgreSQL schema.

Insert query is currently limited to append.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local PostgreSQL installations are not supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local PostgreSQL installations are supported.

The following video shows how to set up [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):

<iframe title="YouTube video player" />

The following video shows how to set up [Azure Database for PostgreSQL](https://azure.microsoft.com/products/postgresql):

<iframe title="YouTube video player" />

* A PostgreSQL instance.

  * [Create an Amazon RDS for PostgreSQL instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html).
  * [Create an Azure Database for PostgreSQL server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-deploy-on-azure-free-account).
  * [Install PostgreSQL locally](https://www.postgresql.org/docs/current/tutorial-install.html).

* The host name and port number for the instance.

  * For Amazon RDS for PostgreSQL, learn how to [get the host name and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html#postgresql-endpoint).
  * For Azure Database for PostgreSQL, learn how to [get the host](https://learn.microsoft.com/azure/postgresql/flexible-server/quickstart-create-server#get-the-connection-information). The port number is `5432`.
  * For local PostgreSQL installations, these values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.
  * For other installation types, see your PostgreSQL provider's documentation.

* Depending on your network security requirements, you might need to allow access to your instance only from specific IP addresses.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

  To learn how to allow these IP address ranges, see your PostgreSQL provider's documentation, for example with
  [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) or
  [Azure Database for PostgreSQL](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-manage-firewall-portal#create-a-firewall-rule-after-server-is-created).

  <Note>
    For Amazon RDS for PostgreSQL, Amazon recommends that you set the instance's **Public access** setting to **No** by default, as this
    approach is more secure. This means that no
    resources can connect to the instance outside of the instance's associated Virtual Private Cloud (VPC) without extra configuration.
    [Learn more](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Hiding).
    [Access an Amazon RDS instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html).

    If you must enable public access, set the instance's **Public access** setting to **Yes**, and then adjust the instance's related
    security group to allow this access.
    [Learn how](https://repost.aws/en/questions/QUxemKa9u5TV6CmLiO-r5prg/lost-public-access-to-aws-rds-postgresql-instance).

    [Troubleshoot issues with connecting to Amazon RDS instances](https://repost.aws/knowledge-center/rds-connectivity-instance-subnet-vpc).
  </Note>

* A database in the instance.

  * For Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, the default database name is `postgres` unless a custom database name was specified during the instance creation process.
  * For local PostgreSQL installations, learn how to [create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* A table in the database. Learn how to [create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

  For the destination connector, the table must have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the fields `id`, `element_id`, `record_id`, `text`, (and `embeddings`, if you are using `pgvector` and generating vector embeddings), as follows.
  `type` is an optional field, but highly recommended.

  If you are using `pgvector` and generating vector embeddings, the number of dimensions (in this example, `1536`) must match the number of dimensions for the associated embedding model that you use in any related Unstructured workflows or ingestion pipelines.

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT
    );
    ```

    ```sql PostgreSQL with pgvector theme={null}
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536)
    );
    ```
  </CodeGroup>

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  For objects in the `metadata` field that Unstructured produces and that you want to store in PostgreSQL, you must create fields in your table's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```

    ```sql PostgreSQL with pgvector  theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "embeddings" is required if you are generating vector embeddings.
    --   If you are generating embeddings, the number of dimensions in "embeddings" 
    --   must match the number of dimensions for the associated embedding model 
    --   that you use in any related Unstructured workflows or ingestion pipelines.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536),
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```
  </CodeGroup>

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  See also:

  * [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
  * [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  The following video shows how to use the `psql` utility to connect to PostgreSQL, list databases, and list and create tables:

  <iframe title="YouTube video player" />

* A user in the database, and a password for the user.

  * For Amazon RDS for PostgreSQL, learn how to [create a user](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Roles.html).
  * For Azure Database for PostgreSQL, learn how to [create a user](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-create-users).
  * For local PostgreSQL installations, learn how to [create a user](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* Database access for the user.

  * For Amazon RDS for PostgreSQL, learn how to [control user access](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Access.html).
  * For Azure Database for PostgreSQL, learn how to [control user access](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For local PostgreSQL installations, learn how to [give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).
  * For other installation types, see your PostgreSQL provider's documentation.

The PostgreSQL connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[postgres]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `PGHOST` - The host name, represented by `--host` (CLI) or `host` (Python).
* `PGPORT` - The port number, represented by `--port` (CLI) or `port` (Python).
* `PGUSER` - The username, represented by `--username` (CLI) or `username` (Python).
* `PGPASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
* `PGDATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Specify which fields to output in the processed data. This can help prevent
  # database record insert issues, where a particular field in the processed data
  # does not match a column in the database table on insert.
  metadata_includes="id,element_id,text,embeddings,type,system,layout_width,\
  layout_height,points,url,version,date_created,date_modified,date_processed,\
  permissions_data,record_locator,category_depth,parent_id,attached_filename,\
  filetype,last_modified,file_directory,filename,languages,page_number,links,\
  page_name,link_urls,link_texts,sent_from,sent_to,subject,section,\
  header_footer_type,emphasized_text_contents,emphasized_text_tags,\
  text_as_html,regex_metadata,detection_class_prob"

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    postgres \
      --username $PGUSER \
      --password $PGPASSWORD \
      --host $PGHOST \
      --port $PGPORT \
      --database $PGDATABASE
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.postgres import (
      PostgresConnectionConfig,
      PostgresAccessConfig,
      PostgresUploaderConfig,
      PostgresUploadStagerConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      # Specify which fields to output in the processed data. This can help prevent
      # database record insert issues, where a particular field in the processed data
      # does not match a column in the database table on insert.
      metadata_includes = [
          "id", "element_id", "text", "embeddings", "type", "system", "layout_width",
          "layout_height", "points", "url", "version", "date_created", "date_modified",
          "date_processed", "permissions_data", "record_locator", "category_depth",
          "parent_id", "attached_filename", "filetype", "last_modified", "file_directory",
          "filename", "languages", "page_number", "links", "page_name", "link_urls",
          "link_texts", "sent_from", "sent_to", "subject", "section", "header_footer_type",
          "emphasized_text_contents", "emphasized_text_tags", "text_as_html", "regex_metadata",
          "detection_class_prob"
      ]

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              metadata_include=metadata_includes,
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=PostgresConnectionConfig(
              access_config=PostgresAccessConfig(password=os.getenv("PGPASSWORD")),
              host=os.getenv("PGHOST"),
              port=os.getenv("PGPORT"),
              username=os.getenv("PGUSER"),
              database=os.getenv("PGDATABASE")
          ),
          stager_config=PostgresUploadStagerConfig(),
          uploader_config=PostgresUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Qdrant
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/qdrant



<NewDocument />

Batch process all your records to store structured outputs in Qdrant.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only [Qdrant Cloud](https://qdrant.tech/documentation/cloud-intro/) is supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Qdrant Cloud,
  [Qdrant local](https://github.com/qdrant/qdrant), and [Qdrant client-server](https://qdrant.tech/documentation/quickstart/) are supported.

The following video shows how to set up Qdrant Cloud:

<iframe title="YouTube video player" />

* For Qdrant local, the path to the local Qdrant installation, for example: `/qdrant/local`

* For Qdrant client-server, the Qdrant server URL, for example: `http://localhost:6333`

* For Qdrant Cloud:

  * A [Qdrant account](https://cloud.qdrant.io/login).

  * A [Qdrant cluster](https://qdrant.tech/documentation/cloud/create-cluster/).

  * The cluster's URL. To get this URL, do the following:

    1. Sign in to your Qdrant Cloud account.
    2. On the sidebar, under **Dashboard**, click **Clusters**.
    3. Click the cluster's name.
    4. Note the value of the **Endpoint** field, for example: `https://<random-guid>.<region-id>.<cloud-provider>.cloud.qdrant.io`.

  * A [Qdrant API key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys).

* The name of the target [collection](https://qdrant.tech/documentation/concepts/collections) on the Qdrant local installation,
  Qdrant server, or Qdrant Cloud cluster.

  Qdrant requires the target collection to exist before Unstructured can write to the collection.
  The following example code demonstrates the use of the [Python Qdrant Client](https://pypi.org/project/qdrant-client/) to create
  a collection on a Qdrant Cloud cluster, configuring the collection for vectors with 3072 dimensions:

  ```python Python theme={null}
  from qdrant_client import QdrantClient, models
  import os

  client = QdrantClient(
      url=os.getenv("QDRANT_URL"),
      api_key=os.getenv("QDRANT_API_KEY")
  )

  client.create_collection(
      collection_name=os.getenv("QDRANT_COLLECTION"),
      vectors_config=models.VectorParams(
          size=3072,
          distance=models.Distance.COSINE
      )
  )

  collection = client.get_collection(
                   collection_name=os.getenv("QDRANT_COLLECTION")
               )

  print(f"The collection named '{os.getenv("QDRANT_COLLECTION")}' exists and " +
        f"has a status of '{collection.status}'.")
  ```

The Qdrant connector dependencies.

```bash theme={null}
pip install "unstructured-ingest[qdrant]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `QDRANT_COLLECTION` - The name of the target collection on the Qdrant local installation,
  Qdrant server, or Qdrant Cloud cluster, represented by `--collection-name` (CLI) or `collection_name` (Python).
* For Qdrant local, `QDRANT_PATH` - The path to the local Qdrant installation, represented by `--path` (CLI) or `path` (Python).
* For Qdrant client-server, `QDRANT_URL` - The Qdrant server's URL, represented by `--url` (CLI) or `url` (Python).
* For Qdrant Cloud:

  * `QDRANT_URL` - The Qdrant cluster's URL, represented by `--url` (CLI) or `url` (Python).
  * `QDRANT_API_KEY` - The Qdrant API key, represented by `--api-key` (CLI) or `api_key` (Python).

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  # For Qdrant local:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-local \
      --path $QDRANT_PATH \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1

  # For Qdrant client-server:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-server \
      --url $QDRANT_URL \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1

  # For Qdrant cloud:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    qdrant-cloud \
      --url $QDRANT_URL \
      --api-key $QDRANT_API_KEY \
      --collection-name $QDRANT_COLLECTION \
      --batch-size 50 \
      --num-processes 1
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # For Qdrant local:
  # from unstructured_ingest.processes.connectors.qdrant.local import (
  #     LocalQdrantConnectionConfig,
  #     LocalQdrantAccessConfig,
  #     LocalQdrantUploadStagerConfig,
  #     LocalQdrantUploaderConfig
  # )

  # For Qdrant client-server:
  # from unstructured_ingest.processes.connectors.qdrant.server import (
  #     ServerQdrantConnectionConfig,
  #     ServerQdrantAccessConfig,
  #     ServerQdrantUploadStagerConfig,
  #     ServerQdrantUploaderConfig
  # )

  # For Qdrant Cloud:
  from unstructured_ingest.processes.connectors.qdrant.cloud import (
      CloudQdrantConnectionConfig,
      CloudQdrantAccessConfig,
      CloudQdrantUploadStagerConfig,
      CloudQdrantUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),

          # For Qdrant local:
          # destination_connection_config=LocalQdrantConnectionConfig(
          #     access_config=LocalQdrantAccessConfig(),
          #     path=os.getenv("QDRANT_PATH")
          # ),
          # stager_config=LocalQdrantUploadStagerConfig(),
          # uploader_config=LocalQdrantUploaderConfig(
          #     collection_name=os.getenv("QDRANT_COLLECTION"),
          #     batch_size=50,
          #     num_processes=1
          # )

          # For Qdrant client-server:
          # destination_connection_config=ServerQdrantConnectionConfig(
          #     access_config=ServerQdrantAccessConfig(),
          #     url=os.getenv("QDRANT_URL")
          # ),
          # stager_config=ServerQdrantUploadStagerConfig(),
          # uploader_config=ServerQdrantUploaderConfig(
          #     collection_name=os.getenv("QDRANT_COLLECTION"),
          #     batch_size=50,
          #     num_processes=1
          # )

          # For Qdrant cloud:
          destination_connection_config=CloudQdrantConnectionConfig(
              access_config=CloudQdrantAccessConfig(
                  api_key=os.getenv("QDRANT_API_KEY")
              ),
              url=os.getenv("QDRANT_URL")
          ),
          stager_config=CloudQdrantUploadStagerConfig(),
          uploader_config=CloudQdrantUploaderConfig(
              collection_name=os.getenv("QDRANT_COLLECTION"),
              batch_size=50,
              num_processes=1
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Redis
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/redis



<NewDocument />

Batch process all your records to store structured outputs in Redis.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Redis](https://redis.io) database, for example in [Redis Cloud](https://redis.io/cloud/).
* The target database's hostname and port number. [Create a database in Redis Cloud](https://redis.io/docs/latest/operate/rc/rc-quickstart/#create-an-account).
* The username and password for the target database. [Get the username and password in Redis Cloud](https://redis.io/docs/latest/operate/rc/rc-quickstart/#connect-to-a-database).
* The database number for the target database. Redis databases are typically numbered from 0 to 15, with the default database number typically being 0.

The Redis connector dependencies.

```bash theme={null}
pip install "unstructured-ingest[redis]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* For connecting with a Redis connection string, `REDIS_URI`, represented by `--uri` (CLI) or `uri` (Python). Redis connection strings use the following format:

  ```
  <protocol>://<username>:<password>@<hostname>:<port>?ssl=<true|false>&db=<db_number>
  ```

* For connecting with Redis connection properties:

  * `REDIS_HOST` - The hostname of the target Redis database, represented by `--host` (CLI) or `host` (Python).
  * `REDIS_PORT` - The database's port number, represented by `--port` (CLI) or `port` (Python).
  * `REDIS_DATABASE` - The database number of the target database, represented by `--database` (CLI) or `database` (Python).
  * `REDIS_USERNAME` - The username for the database, represented by `--username` (CLI) or `username` (Python).
  * `REDIS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
  * `REDIS_KEY_PREFIX` - A string to prepend to each element's `element_id`, represented by `--key-prefix` (CLI) or `key_prefix` (Python).

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  # Chunking and embedding are optional.

  # Use a Redis connection string:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    redis \
      --uri $REDIS_URI,
      --no-ssl \
      --key-prefix $REDIS_KEY_PREFIX \
      --batch-size 100

  # Use Redis connection properties:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    redis \
      --host $REDIS_HOST \
      --port 14453
      --database 0 \
      --username $REDIS_USERNAME \
      --password $REDIS_PASSWORD \
      --no-ssl \
      --key-prefix $REDIS_KEY_PREFIX \
      --batch-size 100
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.redisdb import (
      RedisAccessConfig,
      RedisConnectionConfig,
      RedisUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          
          # Use a Redis connection string.
          # destination_connection_config=RedisConnectionConfig(
          #     access_config=RedisAccessConfig(
          #         uri = os.getenv("REDIS_URI")
          #     )
          # ),

          # Use Redis connection properties.
          destination_connection_config=RedisConnectionConfig(
              access_config=RedisAccessConfig(
                  password=os.getenv("REDIS_PASSWORD")
              ),
              host=os.getenv("REDIS_HOST"),
              database=int(os.getenv("REDIS_DATABASE")),
              port=int(os.getenv("REDIS_PORT")),
              username=os.getenv("REDIS_USERNAME"),
              ssl=False
          ),
          uploader_config=RedisUploaderConfig(
              batch_size=100
              key_prefix=os.getenv("REDIS_KEY_PREFIX")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# S3
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/s3



Batch process all your records to store structured outputs in an S3 bucket.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Amazon S3 requirements:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
  [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

  <img alt="Enable anonymous bucket access" />

* For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
  can be granted in one of the following ways:

  * Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
    To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
    Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

    <iframe title="YouTube video player" />

  * Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
    Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
    [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
    [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
    a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

    <Warning>
      AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
      generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
      You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
      a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

      To overwrite the expired credentials with the new set:

      * For Unstructured Pipelines, manually update the AWS Key, AWS Secret Key, and STS Token fields in Unstructured Pipelines
        for the corresponding S3 [source](/pipelines/sources/s3) or [destination](/pipelines/destinations/s3) connector.
      * For the Unstructured API, use the Unstructured API's workflow operations to call the
        [update source](/api-reference/api/source/update-source) or
        [update destination](/api-reference/api/destination/update-destination) connector operation
        for the corresponding S3 [source](/api-reference/workflow/sources/s3) or
        [destination](/api-reference/workflow/destinations/s3) connector.
      * For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
        corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
    </Warning>

* If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

The S3 connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.

* If the bucket does not have anonymous access enabled, provide the AWS credentials:

  * `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
  * `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
  * `AWS_STS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).

* If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a pipeline that uses S3 as the destination

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    s3 \
      --remote-url $AWS_S3_URL \
      --key $AWS_ACCESS_KEY_ID \
      --secret $AWS_SECRET_ACCESS_KEY \
      --token $AWS_STS_TOKEN # If using AWS STS token.
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.fsspec.s3 import (
      S3ConnectionConfig,
      S3AccessConfig,
      S3UploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=S3ConnectionConfig(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY"),
                  token=os.getenv("AWS_STS_TOKEN", None), # If using AWS STS token.
              )
          ),
          uploader_config=S3UploaderConfig(remote_url=os.getenv("AWS_S3_URL"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SFTP
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/sftp



Batch process all your records to store structured outputs in SFTP.

The requirements are as follows.

* The SFTP server hostname, port, username, and password.

  SFTP servers are offered by several vendors. For example, the following video shows how to create and set up an SFTP server by using AWS Transfer Family:

  <iframe title="YouTube video player" />

* The directory path to start accessing data from, specified as `sftp://<path>/<to>/<directory>`.

See the [SFTP documentation](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[sftp]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `SFTP_HOST` - The SFTP hostname, represented by `--host` (CLI) or `host` (Python).
* `SFTP_PORT` - The SFTP port number, represented by `--port` (CLI) or `port` (Python).
* `SFTP_REMOTE_URL` - The directory path to start accessing data from, represented by `--remote-url` (CLI) or `remote_url` (Python).
* `SFTP_USERNAME` - The SFTP username, represented by `--username` (CLI) or `username` (Python).
* `SFTP_PASSWORD` - The SFTP password, represented by `--password` (CLI) or `password` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
    sftp \
      --host $SFTP_HOST \
      --port $SFTP_PORT \
      --username $SFTP_USERNAME \
      --password $SFTP_PASSWORD \
      --remote-url $SFTP_REMOTE_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.sftp import (
      SftpConnectionConfig,
      SftpAccessConfig,
      SftpUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SftpConnectionConfig(
              access_config=SftpAccessConfig(password=os.getenv("SFTP_PASSWORD")),
              host=os.getenv("SFTP_HOST"),
              port=os.getenv("SFTP_PORT"),
              username=os.getenv("SFTP_USERNAME")
          ),
          uploader_config=SftpUploaderConfig(remote_url=os.getenv("SFTP_REMOTE_URL"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SingleStore
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/singlestore



Batch process all your records to store structured outputs in a SingleStore account.

The requirements are as follows.

* A SingleStore deployment, database, and table. [Learn how](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/).
* The hostname for the SingleStore deployment.
* The port for the host.
* The username for the deployment.
* The password for the user.
* The name of the database in the deployment.
* The name of the table in the database.

To get the values for the hostname, port, username, and password:

1. In your SingleStore account's dashboard sidebar, click **Deployments**.
2. From the drop-down list at the top of the **Deployments** page, select your deployment.
3. On the **Overview** tab, in the **Compute** area, in the **Connect** drop-down list for your deployment, select
   **Your App**.
4. If a **Create User** dialog box appears, note the **User** name and **Password** values.
5. In the **Connect to Workspace** pane's **Your App** tab, note the string in the following format:

```
<user-name>:<password>@<host>:<port>
```

* `<user-name>` is the username.
* `<password>` is the user's password.
* `<host>` is the workspace's hostname.
* `<post>` is the host's port.

To get the values for the database and table names:

1. In your SingleStore dashboard's sidebar, click **Deployments**.
2. From the drop-down list at the top of the **Deployments** page, select your deployment.
3. On the **Databases** tab, note **Name** of your database.
4. Click the database and, on the **Tables** tab, note the name of your table.

The SingleStore connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[singlestore]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `SINGLESTORE_HOST` - The hostname for the SingleStore deployment, represented by `--host` (CLI) or `host` (Python).
* `SINGLESTORE_PORT` - The port for the host, represented by `--port` (CLI) or `port` (Python).
* `SINGLESTORE_USER` - The username for the deployment, represented by `--user` (CLI) or `user` (Python).
* `SINGLESTORE_PASSWORD` - The password for the user, represented by `--password` (CLI) or `password` (Python).
* `SINGLESTORE_DB` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).
* `SINGLESTORE_TABLE` - The name of the table in the database, represented by `--table-name` (CLI) or `table_name` (Python).

The table's schema must match the schema of the documents that Unstructured produces.

During insertion, JSON objects are flattened, and underscores are placed between nested object names.
For example, the following JSON object matches a column in the table named `data_source_date_created`:

```json theme={null}
{
    "...": "...",
    "data_source": {
        "date_created": "1719963233.949"
    },
    "...": "..."
}
```

The `metadata` object itself is also flattened in a similar way. However, for nested objects in `metadata`,
the column in the table does not start with `metadata_`. For example, the following JSON object matches a
column in the table named `last_modified`:

```json theme={null}
{
    "...": "...",
    "metadata": {
        "...": "...",
        "last_modified": "2022-12-16T17:04:16-05:00",
        "...": "..."
    },
    "...": "..."
}
```

Unstructured cannot provide a table schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

You can adapt the following table schema example for your own needs:

```sql theme={null}
CREATE TABLE elements (
    id INT PRIMARY KEY NOT NULL AUTO_INCREMENT,
    record_id TEXT,
    element_id TEXT,
    text TEXT,
    embeddings Vector(384),
    parent_id TEXT,
    page_number TEXT,
    is_continuation BOOLEAN,
    orig_elements TEXT,
    partitioner_type TEXT
);
```

See also:

* [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/)
  in the SingleStore documentation
* [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    singlestore \
      --host $SINGLESTORE_HOST \
      --port $SINGLESTORE_PORT \
      --user $SINGLESTORE_USER \
      --password $SINGLESTORE_PASSWORD \
      --database $SINGLESTORE_DB \
      --table-name $SINGLESTORE_TABLE
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.singlestore import (
      SingleStoreConnectionConfig,
      SingleStoreAccessConfig,
      SingleStoreUploadStagerConfig,
      SingleStoreUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SingleStoreConnectionConfig(
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB"),
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              )
          ),
          stager_config=SingleStoreUploadStagerConfig(drop_empty_cols=True),
          uploader_config=SingleStoreUploaderConfig(table_name=os.getenv("SINGLESTORE_TABLE"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Snowflake
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/snowflake



Batch process all your records to store structured outputs in Snowflake.

The requirements are as follows.

* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.

  <iframe title="YouTube video player" />

  To get the identifier for the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
  3. On the **Account** tab, note the value of the **Account Identifier** field.

  Alternatively, the following Snowflake query returns the current account's identifier:

  ```text theme={null}
  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  ```

* A Snowflake user, which can be a service user (recommended) or a human user.

  To create a service user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Projects > Worksheets**.

  3. Click the **+** button to create a SQL worksheet.

  4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:

     * Replace `<service-user-name>` with some name for the service user.
     * Replace `<default-role-name>` with the name of any default role for the service user to use.

     ```sql theme={null}
     CREATE USER <service-user-name>
     DEFAULT_ROLE = "<default-role-name>"
     TYPE = SERVICE
     ```

  5. Click the arrow icon to run the worksheet, which creates the service user.

  6. To get their login name, on the navigation menu, click **Admin > Users & Roles**.

  7. On the **Users** tab, in the list of available users, click the name of the target user.

  8. In the **About** tile, note the **Login Name** for the user.

  To create a human user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & roles**.
  3. Click the **Users** tab.
  4. Click **+ User**.
  5. Follow the on-screen guidance to specify the user's settings.
  6. Click **Create User**.
  7. To get their login name, on the navigation menu, click **Admin > Users & Roles**.
  8. On the **Users** tab, in the list of available users, click the name of the target user.
  9. In the **About** tile, note the **Login Name** for the user.

* A programmatic access token (PAT) for the Snowflake user.

  <iframe title="YouTube video player" />

  To create a programmatic access token (PAT) for a user:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.

  3. On the **Users** tab, in the list of available users, click the name of the target user.

  4. In the **Programmatic access tokens** tile, click the **Generate new token** button.

  5. Follow the on-screen guidance to specify the PAT's settings.

     <Warning>
       You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later.
       Once this PAT expires, the connector will stop working.
       To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and
       update your connector's settings with your new PAT's value.

       Unstructured does not notify you when a PAT is about to expire or has already expired.
       You are responsible for tracking your PATs' expiration dates and taking corrective action before they expire.
     </Warning>

  6. Click **Generate**.

  7. Copy the generated PAT's value to a secure location, as you will not be able to access it again. If you lose this PAT's value,
     you will need to repeat this procedure to generate a new, replacement one.

  The PAT will not work unless the Snowflake account also has a valid
  [network rule](https://docs.snowflake.com/en/user-guide/network-rules) along with a valid
  [network policy](https://docs.snowflake.com/en/user-guide/network-policies) attached to that rule.
  The network rule must also be activated on the Snowflake account to begin taking effect.

  To create a valid network rule:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Rules**.

  3. Click **+ Network Rule**.

  4. Enter some name for the network rule.

  5. For **Type**, select **IPv4**.

  6. For **Mode**, select **Ingress**.

  7. For **Identifiers**, next to the magnifying glass icon, enter `0.0.0.0/0`, and then press **Enter**.

     <Note>
       The `0.0.0.0/0` value allows all IP addresses to access the Snowflake account.
       You can specify a more specific IP address range if you prefer. However, this more specific IP address range
       will apply to all users, including the user for which you created the PAT.
     </Note>

  8. Click **Create Network Rule**.

  To create a valid network policy, attaching the preceding network rule to this policy at the same time:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click **+ Network Policy**.
  4. Enter some name for the network policy.
  5. Make sure **Allowed** is selected.
  6. In the **Select rule** drop-down list, select the precedingnetwork rule to attach to this network policy.
  7. Click **Create Network Policy**.

  To activate the network rule in the account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click the name of the precedingnetwork policy to activate.
  4. In the policy's side panel, click the ellipsis (three dots) icon, and then click **Activate On Account**.
  5. Click **Activate policy**.

* (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake [user's login name (not username) and the user's password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
  This user must be a human user. Passwords are not supported for service users.

* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  <iframe title="YouTube video player" />

  * To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
  * To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
  * To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
  * To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
  * To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
  3. Click the **Roles** tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  ```text theme={null}
  SHOW ROLES;
  ```

  [Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).

* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

  <iframe title="YouTube video player" />

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:

  ```text theme={null}
  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  ```

* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

  <iframe title="YouTube video player" />

  To view a list of available databases in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  ```text theme={null}
  SHOW DATABASES;
  ```

* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  ```text theme={null}
  SHOW SCHEMAS;
  ```

  The following Snowflake query returns a list of available schemas for the database named `<database_name>` in the current account:

  ```text theme={null}
  SHOW SCHEMAS IN DATABASE <database_name>;
  ```

* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand **Tables**.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named `<schema_name>` in the datbase named
  `<database_name>` in the current account:

  ```text theme={null}
  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  ```

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the columns `ID`, `ELEMENT_ID`, and `RECORD_ID`. The columns `TEXT` and `TYPE` are optional, but highly recommended.
  If you are generating embeddings, then you must also include the column `EMBEDDINGS`.

  In the following `CREATE TABLE` statement, replace the following placeholders with the appropriate values:

  * `<database_name>`: The name of the target database in the Snowflake account.
  * `<schema_name>`: The name of the target schema in the database.
  * `<number-of-dimensions>`: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search,
    this value must match the number of dimensions that you plan to have Snowflake generate or search against.

  ```sql SQL theme={null}
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      ELEMENT_ID VARCHAR,
      RECORD_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>)
  );
  ```

  For objects in the `metadata` field that Unstructured produces and that you want to store in a Snowflake table, you must create columns in your table's schema that
  follow Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  ```sql theme={null}
  -- The ID, RECORD_ID, and ELEMENT_ID columns are required.
  -- TEXT and TYPE are not required but highly recommended.
  -- EMBEDDINGS is required if embeddings are being generated.
  -- All other "metadata" columns are optional.
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      RECORD_ID VARCHAR,
      ELEMENT_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
      IS_EXTRACTED VARCHAR,
      POINTS VARCHAR,
      SYSTEM VARCHAR,
      LAYOUT_WIDTH INTEGER,
      LAYOUT_HEIGHT INTEGER,
      FILETYPE VARCHAR,
      LANGUAGES ARRAY,
      PAGE_NUMBER VARCHAR,
      IMAGE_MIME_TYPE VARCHAR,
      FILENAME VARCHAR,
      URL VARCHAR,
      RECORD_LOCATOR VARCHAR
  );
  ```

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).

The Snowflake connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[snowflake]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `SNOWFLAKE_ACCOUNT` - The ID of the target Snowflake account, represented by `--account` (CLI) or `account` (Python).

* `SNOWFLAKE_USER`  - The name of the target Snowflake user, represented by `--user` (CLI) or `user` (Python).

* `SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN` - The user's programmatic access token (PAT), represented by `--password` (CLI) or `password` (Python).

  <Note>
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  </Note>

* `SNOWFLAKE_ROLE` - The target role for the user, represented by `--role` (CLI) or `role` (Python).

* `SNOWFLAKE_HOST`  - The hostname for the target Snowflake warehouse, represented by `--host` (CLI) or `host` (Python).

* `SNOWFLAKE_PORT`  - The warehouse's port number, represented by `--port` (CLI) or `port` (Python).  The default is `443` if not otherwise specified.

* `SNOWFLAKE_DATABASE`  - The name of the target Snowflake database, represented by `--database` (CLI) or `database` (Python).

* `SNOWFLAKE_SCHEMA` - The name of the target schema in the database, represented by `--schema` (CLI) or `schema` (Python).

* `SNOWFLAKE_TABLE` - The name of the target table in the schema, represented by `--table-name` (CLI) or `table_name` (Python). For the destination connector, the default is `elements` if not otherwise specified.

* `SNOWFLAKE_RECORD_ID_KEY` - The name of the column in the table that uniquely identifies each record, represented by:

  * For the source connector, `--id-column` (CLI) or `id_column` (Python).
  * For the destination connector, `--record-id-key` (CLI) or `record_id_key` (Python). For the destination connector, the default is `record_id` if not otherwise specified.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    snowflake \
      --account $SNOWFLAKE_ACCOUNT \
      --user $SNOWFLAKE_USER \
      --password $SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN \
      --role $SNOWFLAKE_ROLE \
      --host $SNOWFLAKE_HOST \
      --port $SNOWFLAKE_PORT \
      --database $SNOWFLAKE_DATABASE \
      --schema $SNOWFLAKE_SCHEMA \
      --batch-size 50 \
      --table-name $SNOWFLAKE_TABLE \
      --record-id-key $SNOWFLAKE_RECORD_ID_KEY \

  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.snowflake import (
      SnowflakeConnectionConfig,
      SnowflakeAccessConfig,
      SnowflakeUploadStagerConfig,
      SnowflakeUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SnowflakeConnectionConfig(
              access_config=SnowflakeAccessConfig(
                  password=os.getenv("SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN")
              ),
              account=os.getenv("SNOWFLAKE_ACCOUNT"),
              user=os.getenv("SNOWFLAKE_USER"),
              host=os.getenv("SNOWFLAKE_HOST"),
              port=os.getenv("SNOWFLAKE_PORT"),
              database=os.getenv("SNOWFLAKE_DATABASE"),
              schema=os.getenv("SNOWFLAKE_SCHEMA"),
              role=os.getenv("SNOWFLAKE_ROLE")
          ),
          stager_config=SnowflakeUploadStagerConfig(),
          uploader_config=SnowflakeUploaderConfig(
              batch_size=50,
              table_name=os.getenv("SNOWFLAKE_TABLE"),
              record_id_key=os.getenv("SNOWFLAKE_RECORD_ID_KEY")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SQLite
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/sqlite



Batch process all your records to store structured outputs in a SQLite schema.

Insert query is currently limited to append.

The requirements are as follows.

* A SQLite instance. [Download and install SQLite](https://www.sqlitetutorial.net/download-install-sqlite/).
* A SQLite database. [Create a database](https://www.sqlite.org/quickstart.html).
* The path to the database's `.db` file.
* A table in the database. [Create a table](https://www.sqlite.org/cli.html).

  The table's schema must match the schema of the documents that Unstructured produces.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  You can adapt the following table schema example for your own needs:

  ```sql SQLite theme={null}
  CREATE TABLE elements (
      id TEXT PRIMARY KEY,
      record_id TEXT,
      element_id TEXT,
      text TEXT,
      embeddings TEXT,
      parent_id TEXT,
      page_number INTEGER,
      is_continuation INTEGER,
      orig_elements TEXT,
      partitioner_type TEXT
  );
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) for SQLite
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `SQLITE_DB_PATH` - The path to the database's `.db` file, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Specify which fields to output in the processed data. This can help prevent
  # database record insert issues, where a particular field in the processed data
  # does not match a column in the database table on insert.
  metadata_includes="id,element_id,text,embeddings,type,system,layout_width,\
  layout_height,points,url,version,date_created,date_modified,date_processed,\
  permissions_data,record_locator,category_depth,parent_id,attached_filename,\
  filetype,last_modified,file_directory,filename,languages,page_number,links,\
  page_name,link_urls,link_texts,sent_from,sent_to,subject,section,\
  header_footer_type,emphasized_text_contents,emphasized_text_tags,\
  text_as_html,regex_metadata,detection_class_prob"

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    sqlite \
      --database-path $SQLITE_DB_PATH
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.sqlite import (
      SQLiteConnectionConfig,
      SQLiteAccessConfig,
      SQLiteUploaderConfig,
      SQLiteUploadStagerConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      # Specify which fields to output in the processed data. This can help prevent
      # database record insert issues, where a particular field in the processed data
      # does not match a column in the database table on insert.
      metadata_includes = [
          "id", "element_id", "text", "embeddings", "type", "system", "layout_width",
          "layout_height", "points", "url", "version", "date_created", "date_modified",
          "date_processed", "permissions_data", "record_locator", "category_depth",
          "parent_id", "attached_filename", "filetype", "last_modified", "file_directory",
          "filename", "languages", "page_number", "links", "page_name", "link_urls",
          "link_texts", "sent_from", "sent_to", "subject", "section", "header_footer_type",
          "emphasized_text_contents", "emphasized_text_tags", "text_as_html", "regex_metadata",
          "detection_class_prob"
      ]

      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              metadata_include=metadata_includes,
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=SQLiteConnectionConfig(
              access_config=SQLiteAccessConfig(),
              database_path=os.getenv("SQLITE_DB_PATH")
          ),
          stager_config=SQLiteUploadStagerConfig(),
          uploader_config=SQLiteUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Teradata
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/teradata-sql



Batch process all your records to store structured outputs in Teradata.

The requirements are as follows.

* A Teradata Vantage system that can be accessed by its host name or IP address.

  For example, a Teradata Vantage system in Teradata ClearScape Analytics Experience includes:

  * A Teradata ClearScape Analytics Experience account.
  * An environment in the account.
  * A Teradata Vantage database in the environment.
  * The name and password for a Teradata user who has the appropriate access to the database.

  [Learn how to create these in Teradata ClearScape Analytics Experience](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/).

* The system's corresponding host name or IP address.

  For example, you can get these values from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics Experience account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Host** value.<br />

* The name of the target database in the system. To get a list of available databases in the system, you can run a Teradata SQL query such as the following:

  ```sql theme={null}
  SELECT DatabaseName 
  FROM DBC.DatabasesV 
  ORDER BY DatabaseName;
  ```

* The name of the target table in the database. To get a list of available tables in a database, you can run a Teradata SQL query such as the following, replacing `<database-name>` with the name of the target database:

  ```sql theme={null}
  SELECT TableName
  FROM DBC.TablesV
  WHERE DatabaseName = '<database-name>' AND TableKind = 'T'
  ORDER BY TableName;
  ```

  When Unstructured writes rows to a table, the table's columns must have a schema that is compatible with Unstructured.
  Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances.
  This is because these schemas will vary based on
  your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
  any custom post-processing code that you run; and other factors.

  In any case, note the following about table schemas:

  * The following columns are always required by Unstructured: `record_id` and `element_id`.
  * The following columns are optional for Unstructured, but highly recommended: `text` and `type`.
  * The rest of the columns are optional and typically will be output by Unstructured as part of the `metadata` field.
  * If Unstructured is generating vector embeddings, the `embeddings` column is also required.

    <Warning>
      The destination connector outputs Unstructured-generated [embeddings](/concepts/embedding) that are not directly compatible
      with Teradata Enterprise Vector Store. To use embeddings with Teradata Enterprise Vector Store, Unstructured
      recommends that you choose from among the following options:

      * Define a column in your target table named `embeddings` that is of type `VARCHAR(64000)`, to store the
        Unstructured-generated embeddings. After Unstructured adds its embeddings to your `embeddings` column,
        choose from among Teradata's options to convert the `embeddings` column's `VARCHAR` values to the
        Teradata [VECTOR Data Type](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/VECTOR-Data-Type)
        yourself.
      * Omit any columns named `embedding`, `message`, or `num_tokens` from your target table. Then choose from
        among Teradata's options (such as [AI\_TextEmbeddings](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/In-Database-Analytic-Functions-for-Vector-Store))
        to have Teradata generate the embeddings for you, instead of having Unstructured generate them.
    </Warning>

  Here is an example table schema that is compatible with Unstructured. It includes all of the required and recommended columns, as
  well as a few additional columns that are typically output by Unstructured as part of the `metadata` field. Be sure to replace
  `<database-name>` with the name of the target database and `<table-name>` with the name of the target table (by Unstructured convention,
  the table name is typically `elements`, but this is not a requirement).

  ```sql theme={null}
  CREATE SET TABLE "<database-name>"."<table-name>" (
    "id" VARCHAR(64) NOT NULL,
    PRIMARY KEY ("id"),
    "record_id" VARCHAR(64),
    "element_id" VARCHAR(64),
    "text" VARCHAR(32000) CHARACTER SET UNICODE,
    "type" VARCHAR(50),
    "embeddings" VARCHAR(64000), -- Add this column only if Unstructured is generating vector embeddings.
    "last_modified" VARCHAR(50),
    "languages" VARCHAR(200),
    "file_directory" VARCHAR(500),
    "filename" VARCHAR(255),
    "filetype" VARCHAR(50),
    "record_locator" VARCHAR(1000),
    "date_created" VARCHAR(50),
    "date_modified" VARCHAR(50),
    "date_processed" VARCHAR(50),
    "permissions_data" VARCHAR(1000),
    "filesize_bytes" INTEGER,
    "parent_id" VARCHAR(64)
  )
  ```

* For the source connector, the name of the primary key column in the table (for example, a column named `id`, typically defined as `"id" VARCHAR(64) NOT NULL, PRIMARY KEY ("id")`).

* For the source connector, the names of any specific columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* For the destination connector, the name of the column in the table that uniquely identifies each record for Unstructured to perform any necessary record updates. By default convention, Unstructured expects this field to be named `record_id`.

* The name of the Teradata user who has the appropriate access to the target database.

  For example, you can get this from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Username** value.<br />

  The Teradata SQL command to get a list of available users is as follows:

  ```sql theme={null}
  SELECT UserName 
  FROM DBC.UsersV 
  ORDER BY UserName;
  ```

* The password for the user, which was set up when the user was created.

  If the user has forgotten their password, the Teradata SQL command to change a user's password is as follows, replacing `<user-name>` with the name of the user and `<new-password>` with the new password:

  ```sql theme={null}
  MODIFY USER <user-name> SET PASSWORD = '<new-password>';
  ```

  To change a user's password, you must be an administrator (such as the `DBC` user or another user with `DROP USER` privileges).

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

The Teradata connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[teradata]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `TERADATA_HOST` - The host name, represented by `--host` (CLI) or `host` (Python).
* `TERADATA_PORT` - The port number, represented by `--dbs-port` (CLI) or `dbs_port` (Python). This is optional, and the default is `1025` if not otherwise specified.
* `TERADATA_USERNAME` - The name of the user who has access to the database, represented by `--user` (CLI) or `user` (Python).
* `TERADATA_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
* `TERADATA_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python). If not otherwise specified, the default database name is used. To get the name of the default database, you can run the Teradata SQL command `SELECT DATABASE;`.
* `TERADATA_TABLE` - The name of the table, represented by `--table-name` (CLI) or `table_name` (Python).
* `TERADATA_ID_COLUMN` - For the source connector, the name of the column that uniquely identifies each record in the table, represented by `--id-column` (CLI) or `id_column` (Python).
* `TERADATA_RECORD_ID_KEY` - For the destination connector, the name of the column in the table that uniquely identifies each record, represented by `--record-id-key` (CLI) or `record_id_key` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --strategy fast \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --metadata-include "$metadata_includes" \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    teradata \
      --host $TERADATA_HOST \
      # --dbs-port $TERADATA_PORT \
      --database $TERADATA_DATABASE \
      --user $TERADATA_USERNAME \
      --password $TERADATA_PASSWORD \
      --table-name $TERADATA_TABLE \
      --record-id-key $TERADATA_RECORD_ID_KEY \
      --batch-size 50
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.teradata import(
      TeradataConnectionConfig,
      TeradataAccessConfig,
      TeradataUploaderConfig,
      TeradataUploadStagerConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=TeradataConnectionConfig(
              access_config=TeradataAccessConfig(
                  password=os.getenv("TERADATA_PASSWORD")
              ),
              host=os.getenv("TERADATA_HOST"),
              # dbs_port=os.getenv("TERADATA_PORT"),
              database=os.getenv("TERADATA_DATABASE"),
              user=os.getenv("TERADATA_USERNAME")
          ),
          stager_config=TeradataUploadStagerConfig(),
          uploader_config=TeradataUploaderConfig(
              batch_size=50,
              table_name=os.getenv("TERADATA_TABLE"),
              record_id_key=os.getenv("TERADATA_RECORD_ID_KEY")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Vectara
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/vectara



<NewDocument />

Batch process all your records to store structured outputs in Vectara.

The requirements are as follows.

* A [Vectara account](https://console.vectara.com/signup).
* The [customer ID](https://docs.vectara.com/docs/console-ui/vectara-console-overview#view-the-customer-id) for the account.
* The name and key for the target [corpus](https://docs.vectara.com/docs/console-ui/creating-a-corpus) in the account.
* The [OAuth authentication URL, client ID, and client secret](https://docs.vectara.com/docs/console-ui/app-clients) for accessing the target corpus.

The Vectara connector dependencies.

```bash theme={null}
pip install "unstructured-ingest[vectara]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `VECTARA_CUSTOMER_ID` - The customer ID for the target Vectara account, represented by `--customer-id` (CLI) or `customer_id` (Python).
* `VECTARA_CORPUS_NAME` - The name of the target corpus in the account, represented by `--corpus-name` (CLI) or `corpus_name` (Python).
* `VECTARA_CORPUS_KEY` - The name of the corpus's key, represented by `--corpus-key` (CLI) or `corpus_key` (Python).
* `VECTARA_OAUTH_TOKEN_URL` - The OAuth token URL for getting and refreshing OAuth access tokens in the account, represented by `--token-url` (CLI) or `token_url` (Python).
* `VECTARA_OAUTH_CLIENT_ID` - A valid OAuth client ID in the account, represented by `--oauth-client-id` (CLI) or `oauth_client_id` (Python).
* `VECTARA_OAUTH_CLIENT_SECRET` - The OAuth client secret for the client ID, represented by `--oauth-client-secret` (CLI) or `oauth_client_secret` (Python).

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported.

This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding is optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    vectara \
      --customer-id $VECTARA_CUSTOMER_ID \
      --corpus-name $VECTARA_CORPUS_NAME \
      --corpus-key $VECTARA_CORPUS_KEY \
      --oauth-client-id $VECTARA_OAUTH_CLIENT_ID \
      --oauth-secret $VECTARA_OAUTH_CLIENT_SECRET \
      --token-url $VECTARA_OAUTH_TOKEN_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.vectara import (
      VectaraAccessConfig,
      VectaraConnectionConfig,
      VectaraUploadStagerConfig,
      VectaraUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalConnectionConfig,
      LocalDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=VectaraConnectionConfig(
              access_config=VectaraAccessConfig(
                  oauth_client_id=os.getenv("VECTARA_OAUTH_CLIENT_ID"),
                  oauth_secret=os.getenv("VECTARA_OAUTH_CLIENT_SECRET")
              ),
              customer_id=os.getenv("VECTARA_CUSTOMER_ID"),
              corpus_name=os.getenv("VECTARA_CORPUS_NAME"),
              corpus_key=os.getenv("VECTARA_CORPUS_KEY"),
              token_url=os.getenv("VECTARA_OAUTH_TOKEN_URL")
          ),
          stager_config=VectaraUploadStagerConfig(),
          uploader_config=VectaraUploaderConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Weaviate
Source: https://docs.unstructured.io/open-source/ingestion/destination-connectors/weaviate



Batch process all your records to store structured outputs in a Weaviate database.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview): only [Weaviate Cloud](https://weaviate.io/developers/wcs) clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview): Weaviate Cloud clusters,
  [Weaviate installed locally](https://weaviate.io/developers/weaviate/quickstart/local),
  and [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded) are supported.
* For Weaviate installed locally, you will need the name of the target collection on the local instance.
* For Embedded Weaviate, you will need the instance's connection URL and the name of the target collection on the instance.
* For Weaviate Cloud, you will need:

  <iframe title="YouTube video player" />

  * A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
    [Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).
  * The URL and API key for the database cluster. [Get the URL and API key](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).
  * The name of the target collection in the database. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).

    An existing collection is not required. At runtime, the collection behavior is as follows:

    For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview):

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your Weaviate cluster. If Unstructured generates embeddings,
      the new collection's name will be `U<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `U<short-workflow-id`.

    For [Unstructured Ingest](/open-source/ingestion/overview):

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your Weaviate cluster. The new collection's name will be `Unstructuredautocreated`.

    If Unstructured creates a new collection and generates embeddings, you will not see an `embeddings` property in tools such as the Weaviate Cloud
    **Collections** user interface. To view the generated embeddings, you can run a Weaviate GraphQL query such as the following. In this query, replace `<collection-name>` with
    the name of the new collection, and replace `<property-name>` with the name of each additional available property that
    you want to return results for, such as `text`, `type`, `element_id`, `record_id`, and so on. The embeddings will be
    returned in the `vector` property.

    ```text theme={null}
    {
      Get {
        <collection-name> {
          _additional {
            vector
          }
          <property-name>
          <property-name>
        }
      }
    }
    ```

If [auto-schema](https://docs.weaviate.io/weaviate/config-refs/collections#auto-schema) is enabled in Weaviate (which it is by default),
Weaviate can infer missing properties and add them to the collection definition at run time. However, it is a Weaviate best practice to manually define as much
of the data schema in advance as possible, since manual definition gives you the most control.

The minimum viable schema for Unstructured includes only the `element_id` and `record_id` properties. The `text` and `type` properties should also be included, but they are technically optional.
If you are using Unstructured to generate embeddings, you must

The following code example shows how to use the [weaviate-client](https://pypi.org/project/weaviate-client/) Python package to create a
collection in a Weaviate Cloud database cluster with this minimum viable schema, and to specify that Unstructured will generate the embeddings for this collection.
To connect to a locally hosted Weaviate instance instead, call [weaviate.connect\_to\_local](https://docs.weaviate.io/weaviate/connections/connect-local).
To connect to Embedded Weaviate instead, call [weaviate.connect\_to\_embedded](https://docs.weaviate.io/weaviate/connections/connect-embedded).

```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
```

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

For objects in the `metadata` field that Unstructured produces and that you want to store in a Weaviate collection, be sure to follow
Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You could create corresponding properties in your collection's schema by using the following property names and data types:

```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
        wvc.Property(
            name="metadata",
            data_type=wvc.DataType.OBJECT,
            nested_properties=[
                wvc.Property(name="is_extracted", data_type=wvc.DataType.TEXT),
                wvc.Property(
                    name="coordinates",
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(name="points", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="system", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="layout_width", data_type=wvc.DataType.NUMBER),
                        wvc.Property(name="layout_height", data_type=wvc.DataType.NUMBER),
                    ],
                ),
                wvc.Property(name="filetype", data_type=wvc.DataType.TEXT),
                wvc.Property(name="languages", data_type=wvc.DataType.TEXT_ARRAY),
                wvc.Property(name="page_number", data_type=wvc.DataType.TEXT),
                wvc.Property(name="image_mime_type", data_type=wvc.DataType.TEXT),
                wvc.Property(name="filename", data_type=wvc.DataType.TEXT),
                wvc.Property(
                    name="data_source",
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(name="url", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="record_locator", data_type=wvc.DataType.TEXT),
                    ],
                ),
                wvc.Property(
                    name="entities", 
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(
                            name="items", 
                            data_type=wvc.DataType.OBJECT_ARRAY,
                            nested_properties=[
                                wvc.Property(name="entity", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="type", data_type=wvc.DataType.TEXT),
                            ],
                        ),
                        wvc.Property(
                            name="relationships", 
                            data_type=wvc.DataType.OBJECT_ARRAY,
                            nested_properties=[
                                wvc.Property(name="to", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="from", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="relationship", data_type=wvc.DataType.TEXT),
                            ],
                        ),
                    ],
                ),
            ],
        ),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
```

Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

See also:

* [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
* [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

The Weaviate connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[weaviate]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* For Weaviate installed locally, `WEAVIATE_COLLECTION` - The name of the target collection in the instance, represented by `--collection` (CLI) or `collection` (Python).

* For Embedded Weaviate:

  * `WEAVIATE_HOST` - The connection URL to the instance, represented by `--hostname` (CLI) or `hostname` (Python).
  * `WEAVIATE_COLLECTION` - The name of the target collection in the instance, represented by `--collection` (CLI) or `collection` (Python).
    If no value is provided, see the beginning of this article for the behavior at run time.

* For Weaviate Cloud:

  * `WEAVIATE_CLUSTER_URL` - THE REST endpoint for the Weaviate database cluster, represented by `--cluster-url` (CLI) or `cluster_url` (Python).

  * `WEAVIATE_API_KEY` - The API key for the database cluster, represented by `--api-key` (CLI) or `api_key` (Python).

    <Note>For the CLI, the `--api-key` option here is part of the `weaviate-cloud` command. For Python, the `api_key` parameter here is part of the `CloudWeaviateAccessConfig` object.</Note>

  * `WEAVIATE_COLLECTION` - The name of the target collection in the database, represented by `--collection` (CLI) or `collection` (Python).
    If no value is provided, see the beginning of this article for the behavior at run time.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The source connector can be any of the ones supported. This example uses the local source connector:

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding is optional.

  # For Weaviate installed locally:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    weaviate-local \
      --collection $WEAVIATE_COLLECTION

  # For Embedded Weaviate:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    weaviate-embedded \
      --hostname $WEAVIATE_HOST \
      --collection $WEAVIATE_COLLECTION

  # For Weaviate Cloud:
  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    weaviate-cloud \
      --cluster-url $WEAVIATE_URL \
      --api-key $WEAVIATE_API_KEY \
      --collection $WEAVIATE_COLLECTION
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # For Weaviate installed locally:
  # from unstructured_ingest.processes.connectors.weaviate.local import (
  #     LocalWeaviateConnectionConfig,
  #     LocalWeaviateAccessConfig,
  #     LocalWeaviateUploadStagerConfig,
  #     LocalWeaviateUploaderConfig
  # )

  # For Embedded Weaviate:
  # from unstructured_ingest.processes.connectors.weaviate.embedded import (
  #     EmbeddedWeaviateConnectionConfig,
  #     EmbeddedWeaviateAccessConfig,
  #     EmbeddedWeaviateUploadStagerConfig,
  #     EmbeddedWeaviateUploaderConfig
  # )

  # For Weaviate Cloud:
  from unstructured_ingest.processes.connectors.weaviate.cloud import (
      CloudWeaviateConnectionConfig,
      CloudWeaviateAccessConfig,
      CloudWeaviateUploaderConfig,
      CloudWeaviateUploadStagerConfig
  )

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(
              embedding_provider="openai",
              embedding_model_name=os.getenv("EMBEDDING_MODEL_NAME"),
              embedding_api_key=os.getenv("OPENAI_APIKEY")
          ),

          # For Weaviate installed locally:
          # destination_connection_config=LocalWeaviateConnectionConfig(
          #     access_config=LocalWeaviateAccessConfig()
          # ),
          # stager_config=LocalWeaviateUploadStagerConfig(),
          # uploader_config=LocalWeaviateUploaderConfig(
          #     collection=os.getenv("WEAVIATE_COLLECTION")
          # )

          # For Embedded Weaviate:
          # destination_connection_config=EmbeddedWeaviateConnectionConfig(
          #     access_config=EmbeddedWeaviateAccessConfig(),
          #     hostname=os.getenv("WEAVIATE_HOST")
          # ),
          # stager_config=EmbeddedWeaviateUploadStagerConfig(),
          # uploader_config=EmbeddedWeaviateUploaderConfig(
          #     collection=os.getenv("WEAVIATE_COLLECTION")
          # )

          # For Weaviate Cloud:
          destination_connection_config=CloudWeaviateConnectionConfig(
              access_config=CloudWeaviateAccessConfig(
                  api_key=os.getenv("WEAVIATE_API_KEY")
              ),
              cluster_url=os.getenv("WEAVIATE_CLUSTER_URL")
          ),
          stager_config=CloudWeaviateUploadStagerConfig(),
          uploader_config=CloudWeaviateUploaderConfig(
              collection=os.getenv("WEAVIATE_COLLECTION")
          )
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Process files in batches by using the Unstructured Ingest CLI
Source: https://docs.unstructured.io/open-source/ingestion/ingest-cli



The Unstructured Ingest CLI enables you to use command-line scripts to send files in batches to Unstructured for processing, and to tell Unstructured where to deliver the processed data. [Learn more](/open-source/ingestion/overview#unstructured-ingest-cli).

<Note>
  The Unstructured Ingest CLI does not work with the Unstructured API.

  For information about the Unstructured API, see the [Unstructured API Overview](/api-reference/workflow/overview).
</Note>

## Getting started

You can use the Unstructured Ingest CLI to process files locally, or you can use the Ingest CLI to send files in batches to Unstructured for processing.

Local processing does not use an Unstructured API key or API URL.

Using the Ingest CLI to send files in batches to Unstructured for processing is more robust, and usage is billed to you on a pay-as-you-go basis. Usage requires an Unstructured API key and API URL, as follows:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

3) The Unstructured API URL for Unstructured Ingest was provided to you when your Unstructured account was created.
   If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  The default URL for Unstructured Ingest is the same as the default URL for the legacy Unstructured Partition Endpoint: `https://api.unstructuredapp.io/general/v0/general`.
  However, you should always use the URL that was provided to you when your Unstructured account was created.
</Note>

## Installation

One approach to get started quickly with the Unstructured Ingest CLI is to install Python and then run the following command:

```bash theme={null}
pip install unstructured-ingest
```

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

For additional installation options, see [Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) in the [Ingest](/open-source/ingestion/overview) section.

<Info>To migrate from older, deprecated versions of the Ingest CLI that used `pip install unstructured`, see the [migration guide](/open-source/ingestion/overview#migration-guide).</Info>

## Usage

To call the Unstructured Ingest CLI, follow this calling pattern, where:

* `<source>` is the command name for one of the available [source](/open-source/ingestion/source-connectors/overview) (input) connectors, such as `local` for a local source location, `azure` for an Azure Storage account source, `s3` for an Amazon S3 bucket source, and so on.
* `<destination>` is the command name for one of the available [destination](/open-source/ingestion/destination-connectors/overview) (output) connectors, such as `local` for a local destination, `azure` for an Azure Storage account destination, `s3` for an Amazon S3 bucket destination, and so on.
* `<setting>` is one or more command-line options for specifying how and where Unstructured will ingest the files from the `<source>`, or how and where to deliver the processed data to the `<destination>`.

```bash CLI theme={null}
#!/usr/bin/env bash

unstructured-ingest \
  <source> \
    --<setting1> <value1> \
    --<setting2> <value2> \
    --<settingN> <valueN> \
  <destination> \
    --<setting1> <value1> \
    --<setting2> <value2> \
    --<settingN> <valueN>
```

To learn how to use the Unstructured Ingest CLI to work with a specific source (input) and destination (output) location, see the CLI script examples for the [source](/open-source/ingestion/source-connectors/overview) and [destination](/open-source/ingestion/destination-connectors/overview) connectors that are available for you to choose from.

See also the [ingest configuration](/open-source/ingestion/ingest-configuration/overview) settings for command-line options that enable you to further control how batches are sent and processed.


# Chunking configuration
Source: https://docs.unstructured.io/open-source/ingestion/ingest-configuration/chunking-configuration



A common chunking configuration is a critical element in the data processing pipeline, particularly when creating embeddings and populating vector databases with the results. This configuration defines the parameters governing the segmentation of text into meaningful chunks, whether at the document, paragraph, or sentence level. It plays a pivotal role in determining the size and structure of these chunks, ensuring that they align with the specific requirements of downstream tasks, such as embedding generation and vector database population. By carefully configuring chunking parameters, users can optimize the granularity of data segments, ultimately contributing to more cohesive and contextually rich results. This is crucial for tasks like natural language processing and text analysis, as well as for the efficient storage and retrieval of embeddings in vector databases, enhancing the quality and relevance of the results.

## Configs

* `chunk_api_key`: If `chunk_by_api` is set to `True`, requests that are sent to the Unstructured API will use this Unstructured API key to make authenticated calls.

* `chunk_by_api`: Default: `False`. If set to `True`, uses Unstructured to run chunking. If set to `False`, runs chunking locally.

* `chunk_combine_text_under_n_chars`: Combine consecutive chunks when the first does not exceed length `n` and the second will fit without exceeding the hard-maximum length. Only operative for the `by_title` chunking strategy.

* `chunk_include_orig_elements`: `True` when chunking to add the original elements consolidated to form each chunk to `.metadata.orig_elements` on that chunk.

* `chunk_max_characters`: Default: `500`. The hard-maximum chunk length. No chunk will exceed this length. An oversized element will be divided by text-splitting to fit this window.

* `chunk_multipage_selections`: `True` to ignore page boundaries when chunking such that elements from two different pages can appear in the same chunk. Only operative for the `by_title` chunking strategy.

* `chunk_new_after_n_chars`: The soft-maximum chunk length. Another element will not be added to a chunk of `n` length even when it would fit without exceeding the hard-maximum length.

* `chunk_overlap`: Default: `0`. Prefix each chunk's text with the last overlap of `n` characters from the prior chunk. Only applies to oversized chunks divided by text-splitting. To apply overlap to non-oversized chunks, use `chunk_overlap_all`.

* `chunk_overlap_all`: Applies overlap to chunks formed from whole elements as well as those formed by text-splitting oversized elements. The overlap length is taken from the `chunk_overlap` value.

* `chunking_endpoint`: If `chunk_by_api` is set to `True`, chunking requests are sent to this Unstructured API URL. By default, this URL is the legacy Unstructured Partition Endpoint URL: `https://api.unstructuredapp.io/general/v0/general`. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

* `chunking_strategy`: One of `basic` or `by_title`. When omitted, no chunking is performed. The `basic` strategy maximally fills each chunk with whole elements, up the specified size limits as specified by `max_characters` and `new_after_n_chars`. A single element that exceeds this length is divided into two or more chunks using text-splitting. A `Table` element is never combined with any other element and appears as a chunk of its own or as a sequence of `TableChunk` elements splitting is required. The `by_title` behaviors are the same except that section and optionally page boundaries are respected such that two consecutive elements from different sections appear in separate chunks.

<Note>Chunking will fail if you set both `partition_by_api` to `False` and `chunking_strategy` to `by_page` or `by_similarity`. However, the rest of your data processing pipeline should be unaffected by this setting.</Note>


# Embedding configuration
Source: https://docs.unstructured.io/open-source/ingestion/ingest-configuration/embedding-configuration



A common embedding configuration is a critical component that allows for dynamic selection of embedders and their associated parameters to create vectors from data. This configuration provides the flexibility to choose from various embedding models and fine-tune parameters to optimize the quality and characteristics of the resulting vectors. It enables users to tailor the embedding process to the specific needs of their data and downstream applications, ensuring that the generated vectors effectively capture semantic relationships and contextual information within the dataset.

The core [Unstructured](https://github.com/Unstructured-IO/unstructured) open source library does not enable the generation of embeddings by default. (However you can generate embeddings as a separate step manually.
[Learn how](/open-source/core-functionality/embedding).)

You can configure the Unstructured CLI and Unstructured Ingest Python library to generate embeddings by specifying the
`embedding_provider`, `embedding_api_key`, and `embedding_model_name` options (and, for Amazon Bedrock, additional options), as follows.
You must provide your own API key for the specified embedding provider. To get this API key, you must first create an account with that
provider and set up billing directly with them. You are responsible for all costs associated with using that provider.

Calls to the Unstructured CLI or Unstructured Ingest Python library that are routed to Unstructured's software-as-a-service (SaaS)
for processing (for example, by specifying an Unstructured API key and an Unstructured SaaS URL) require an Unstructured account for billing purposes. Unstructured's costs for generating
embeddings are already included in its account pricing plans. To
generate embeddings, you must specify the `embedding_provider` and `embedding_model_name` configuration options as follows. Unstructured uses its own internal API key
when using the specified provider to generate the embeddings. These
`embedding_provider` and `embedding_model_name` options are limited only to the provider and model names that are supported by the Unstructured API.
[See the list of supported provider and model names](/api-reference/workflow/nodes/embedder/embedder). [Learn how to specify these options](/open-source/how-to/embedding).

## Configs

* `embedding_provider`: The embedding provider to use while doing embedding. Available values include `bedrock`, `azure-openai`, `huggingface`, `mixedbread-ai`, `octoai`, `openai`, `togetherai`, `vertexai`, and `voyageai`.

* `embedding_api_key`: The API key to use, if one is required to generate the embeddings through an API service, such as OpenAI.

* `embedding_aws_access_key_id`: The AWS access key ID to be used for AWS-based embedders, such as Amazon Bedrock.

* `embedding_aws_region`: The AWS Region ID to be used for AWS-based embedders, such as Amazon Bedrock.

* `embedding_aws_secret_access_key`: The AWS secret access key to be used for AWS-based embedders, such as Amazon Bedrock.

* `embedding_model_name`: The specific model to use for the embedding provider, if necessary.

The default `embedding_model_name` values unless otherwise specified are:

* `bedrock`: None

* `azure-openai`: `text-embedding-ada-002`, with 1536 dimensions

* `huggingface`: `sentence-transformers/all-MiniLM-L6-v2`, with 384 dimensions

* `mixedbread-ai`: `mixedbread-ai/mxbai-embed-large-v1`, with 1024 dimensions

* `octoai`: `thenlper/gte-large`, with 1024 dimensions

* `openai`: `text-embedding-ada-002`, with 1536 dimensions

* `togetherai`: `togethercomputer/m2-bert-80M-32k-retrieval`, with 768 dimensions

* `vertexai`: `text-embedding-05`, with 768 dimensions

* `voyageai`: None


# Overview
Source: https://docs.unstructured.io/open-source/ingestion/ingest-configuration/overview



The configurations listed in this section are common to all connectors.
These configurations act as instructions for how to collect, process, and store data.

<iframe title="YouTube video player" />

If you're looking for parameters specific to a connector (such as authentication options) see the [source code](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/unstructured_ingest/processes/connectors).


# Partition configuration
Source: https://docs.unstructured.io/open-source/ingestion/ingest-configuration/partition-configuration



A standard partition configuration is a collection of parameters designed to oversee document partitioning, whether executed through API integration or by the unstructured library on a local system. These parameters serve a dual role, encompassing those passed to the partition method for the initial segmentation of documents and those responsible for coordinating data after processing, including the dynamic metadata associated with each element.

## Configs for Partitioning

* `additional_partition_args`: A JSON string representation of any values to pass through to the `partition` function.

* `encoding`: The encoding method used to decode the text input. By default, UTF-8 will be used.

* `ocr_languages`: The languages present in the document, for use in partitioning, OCR, or both. Multiple languages indicate that the text could be in any of the specified languages.

* `skip_infer_table_types`: List of document types that you want to skip table extraction with.

* `strategy`: Default: `auto`. The strategy to use for partitioning PDF and image files. Uses a layout detection model if set to `hi_res`. Otherwise, partitioning simply extracts the text from the document and processes it.

## Configs for the Process

* `api_key`: If `partition_by_api` is set to `True`, requests that are sent to the Unstructured API will use this Unstructured API key to make authenticated calls.

* `fields_include`: Fields to include in the output JSON. By default, the following fields are included: `element_id`, `text`, `type`, `metadata`, and `embeddings`.

* `flatten_metadata`: Default: `False`. If set to `True`, the hierarchical metadata structure is flattened to have all values exist at the top level.

* `hi_res_model_name`: DEPRECATED. The model to use when `strategy` is set to `hi_res`. By not specifying, you'll get the best object detection model by default.

* `metadata_exclude`: Values from the `metadata` field to exclude from the output.

* `metadata_include`: If provided, only the specified fields are preserved in the `metadata` output.

* `partition_by_api`: Default: `False`. If set to `True`, uses Unstructured to run partitioning. If set to `False`, runs partitioning locally.

* `partition_endpoint`: If `partition_by_api` is set to `True`, partitioning requests are sent to this Unstructured API URL.


# Processor configuration
Source: https://docs.unstructured.io/open-source/ingestion/ingest-configuration/processor-configuration



A common process configuration plays a pivotal role in overseeing the entire ingest process, encompassing various aspects to ensure efficiency and reliability. This configuration includes parameters for managing a pool of workers, which allows for parallelization of the ingest process to maximize throughput. Additionally, it addresses the storage and retrieval of intermediate results, supporting caching strategies that enable replayability in case of errors or interruptions. By configuring worker pools, the process can efficiently handle multiple tasks concurrently, enhancing performance. Furthermore, saving intermediate results allows for the resumption of the process from a known state, reducing data loss and ensuring robustness in the face of unexpected errors or system failures. This comprehensive configuration promotes an organized and resilient approach to data ingestion, improving overall system reliability and efficiency.

## Configs

* `disable_parallelism`: `True` if the `INGEST_DISABLE_PARALLELISM` environment variable is set to `True` (case-insensitive), otherwise `False` (the default).

* `download_only`: Default: `False`. If set to `True`, downloads any files that are not already present in the connector's specified download directory (`download_dir`), or `work_dir` if `download_dir` is not specified, or the default file path for `work_dir` if `work_dir` is not specified.

* `max_connections`: The maximum of connections allowed when running an asynchronous step in the ingest pipeline.

* `max_docs`: The maximum number of documents to be pulled in for the entire ingest process.

* `num_processes`: Default: `2`. For every step that can use a pool of workers to increase throughput, how many workers to configure in the pool.

* `output_dir`: Where the final results will be located when the process is finished. This is regardless of whether a destination is configured. If a directory is not specified, by default a folder named `structured-output`, relative to the current working directory, is used.

* `preserve_downloads`: When set to `False` (the default), will remove downloaded files after they are successfully processed.

* `raise_on_error`: `False `(default) for any single document that fails in the process, causes the error to be logged but allows for all other documents to proceed in the process. If `True`, causes the entire process to fail and raise the error if any one document fails.

* `re_download`: When set to `False` (the default), will not re-download files if they are already present in the download directory. Setting this to `True` downloads the files even if they exist in the download directory. `True` is useful in cases where for some reason you suspect that the files in the remote location might have changed, and you want to be sure that the file copies in the download directory are the most current ones.

* `reprocess`: Default: `False`. If set to `True`, will ignore all content that may have been cached and rerun each step.

* `tqdm`: When set to `False` (the default), will not show a progress bar.

* `uncompress`: When set to `False` (the default), will not uncompress any archived ZIP or TAR files. If set to `True`, and the specified partitioner provides support for compressed ZIP and TAR files, these files will be uncompressed.

* `verbose`: Default: `False`. If set to `True`, debug logging should be included in the output.

* `work_dir`: The file path for where intermediate results should be saved. If a file path is not set, a default file path will be used relative to the user's home directory (`/home/username/.cache/unstructured/ingest/pipeline` or `C:\Users\username\.cache\unstructured\ingest\pipeline`).


# Ingest dependencies
Source: https://docs.unstructured.io/open-source/ingestion/ingest-dependencies



When you install the [Unstructured Ingest CLI](/open-source/ingestion/ingest-cli) and the
[Unstructured Ingest Python library](/open-source/ingestion/python-ingest) by running the command
`pip install unstructured-ingest` by itself, you get the following by default:

* The [local source connector](/open-source/ingestion/source-connectors/local) and the [local destination connector](/open-source/ingestion/destination-connectors/local).
* Support for the following file types:

| File type |
| --------- |
| `.bmp`    |
| `.eml`    |
| `.heic`   |
| `.html`   |
| `.jpg`    |
| `.jpeg`   |
| `.tiff`   |
| `.png`    |
| `.txt`    |
| `.xml`    |

To add support for additional file types, run the following:

| Command                                   | File type |
| ----------------------------------------- | --------- |
| `pip install "unstructured-ingest[csv]"`  | `.csv`    |
| `pip install "unstructured-ingest[doc]"`  | `.doc`    |
| `pip install "unstructured-ingest[docx]"` | `.docx`   |
| `pip install "unstructured-ingest[epub]"` | `.epub`   |
| `pip install "unstructured-ingest[md]"`   | `.md`     |
| `pip install "unstructured-ingest[msg]"`  | `.msg`    |
| `pip install "unstructured-ingest[odt]"`  | `.odt`    |
| `pip install "unstructured-ingest[org]"`  | `.org`    |
| `pip install "unstructured-ingest[pdf]"`  | `.pdf`    |
| `pip install "unstructured-ingest[ppt]"`  | `.ppt`    |
| `pip install "unstructured-ingest[pptx]"` | `.pptx`   |
| `pip install "unstructured-ingest[rtf]"`  | `.rtf`    |
| `pip install "unstructured-ingest[rst]"`  | `.rst`    |
| `pip install "unstructured-ingest[tsv]"`  | `.tsv`    |
| `pip install "unstructured-ingest[xlsx]"` | `.xlsx`   |

To add support for additional connectors, run the following:

| Command                                                 | Connector type       |
| ------------------------------------------------------- | -------------------- |
| `pip install "unstructured-ingest[airtable]"`           | Airtable             |
| `pip install "unstructured-ingest[astra]"`              | Astra DB             |
| `pip install "unstructured-ingest[azure]"`              | Azure Blob Storage   |
| `pip install "unstructured-ingest[azure-ai-search]"`    | Azure AI Search      |
| `pip install "unstructured-ingest[box]"`                | Box                  |
| `pip install "unstructured-ingest[chroma]"`             | Chroma               |
| `pip install "unstructured-ingest[confluence]"`         | Confluence           |
| `pip install "unstructured-ingest[couchbase]"`          | Couchbase            |
| `pip install "unstructured-ingest[databricks-volumes]"` | Databricks Volumes   |
| `pip install "unstructured-ingest[delta-table]"`        | Delta Tables         |
| `pip install "unstructured-ingest[dropbox]"`            | Dropbox              |
| `pip install "unstructured-ingest[dropbox]"`            | DuckDB, MotherDuck   |
| `pip install "unstructured-ingest[elasticsearch]"`      | Elasticsearch        |
| `pip install "unstructured-ingest[gcs]"`                | Google Cloud Storage |
| `pip install "unstructured-ingest[github]"`             | GitHub               |
| `pip install "unstructured-ingest[gitlab]"`             | GitLab               |
| `pip install "unstructured-ingest[google-drive]"`       | Google Drive         |
| `pip install "unstructured-ingest[jira]"`               | JIRA                 |
| `pip install "unstructured-ingest[kafka]"`              | Apache Kafka         |
| `pip install "unstructured-ingest[milvus]"`             | Milvus               |
| `pip install "unstructured-ingest[mongodb]"`            | MongoDB              |
| `pip install "unstructured-ingest[neo4j]"`              | Neo4j                |
| `pip install "unstructured-ingest[notion]"`             | Notion               |
| `pip install "unstructured-ingest[onedrive]"`           | OneDrive             |
| `pip install "unstructured-ingest[opensearch]"`         | OpenSearch           |
| `pip install "unstructured-ingest[outlook]"`            | Outlook              |
| `pip install "unstructured-ingest[pinecone]"`           | Pinecone             |
| `pip install "unstructured-ingest[postgres]"`           | PostgreSQL, SQLite   |
| `pip install "unstructured-ingest[qdrant]"`             | Qdrant               |
| `pip install "unstructured-ingest[redis]"`              | Redis                |
| `pip install "unstructured-ingest[s3]"`                 | Amazon S3            |
| `pip install "unstructured-ingest[sharepoint]"`         | SharePoint           |
| `pip install "unstructured-ingest[salesforce]"`         | Salesforce           |
| `pip install "unstructured-ingest[singlestore]"`        | SingleStore          |
| `pip install "unstructured-ingest[snowflake]"`          | Snowflake            |
| `pip install "unstructured-ingest[sftp]"`               | SFTP                 |
| `pip install "unstructured-ingest[slack]"`              | Slack                |
| `pip install "unstructured-ingest[teradata]"`           | Teradata             |
| `pip install "unstructured-ingest[vectara]"`            | Vectara              |
| `pip install "unstructured-ingest[weaviate]"`           | Weaviate             |

To add support for available embedding libraries, run the following:

| Command                                                 | Embedding library type |
| ------------------------------------------------------- | ---------------------- |
| `pip install "unstructured-ingest[bedrock]"`            | Amazon Bedrock         |
| `pip install "unstructured-ingest[embed-huggingface]"`  | Hugging Face           |
| `pip install "unstructured-ingest[embed-octoai]"`       | OctoAI                 |
| `pip install "unstructured-ingest[embed-vertexai]"`     | Google Vertex AI       |
| `pip install "unstructured-ingest[embed-voyageai]"`     | Voyage AI              |
| `pip install "unstructured-ingest[embed-mixedbreadai]"` | Mixedbread             |
| `pip install "unstructured-ingest[openai]"`             | OpenAI, Azure OpenAI   |
| `pip install "unstructured-ingest[togetherai]"`         | together.ai            |

For details about the specific dependencies that are installed, see:

* [Common dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/common)
* [File type dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/local_partition)
* [Connector dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/connectors)
* [Embedding library dependencies](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/requirements/embed)

See also [setup.py](https://github.com/Unstructured-IO/unstructured-ingest/blob/main/setup.py).


# Overview
Source: https://docs.unstructured.io/open-source/ingestion/overview



<Note>
  Unstructured recommends that you use the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview) instead of the
  Unstructured Ingest CLI or the Unstructured Ingest Python library.

  Unstructured Pipelines and the API provide a full range of partitioning, chunking, embedding, and enrichment options for your files and data.
  It also uses the latest and highest-performing models on the market today, and it has built-in logic to deliver the highest quality results
  at the lowest cost.

  The Unstructured Ingest CLI and the Unstructured Ingest Python library are not being actively updated to include these and other Unstructured API features.
</Note>

You can send multiple files in batches to be ingested by Unstructured for processing.
*Ingestion* is the term that Unstructured uses to refer to the set of activities that happens when files are input for processing. [Learn more](/open-source/ingestion/overview).

You can send batches to Unstructured by using the following tools:

* The [Unstructured Ingest CLI](/open-source/ingestion/ingest-cli), which is part of and builds upon the [Unstructured open source library](/open-source/introduction/overview).
* The [Unstructured Ingest Python](/open-source/ingestion/python-ingest) library, which is also part of and builds upon the [Unstructured open source library](/open-source/introduction/overview).

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured for processing:

<iframe title="YouTube video player" />

The following 5-minute video goes into more detail about the various components of the Unstructured Ingest Python library:

<iframe title="YouTube video player" />

## Ingest flow

The Unstructured ingest flow is similar to an extract, transform and load (ETL) data pipeline.
Because of this, a customer-defined implementation of the Unstructured ingest flow is sometimes referred to as an *ingest pipeline* or simply a *pipeline*.
An Unstructured ingest pipeline contains the following logical steps:

<Steps>
  <Step title="Index">
    Reaches out to the source location and pulls in metadata for each document.

    For example, this could include information such as the path to the files to be analyzed.

    * For the Unstructured CLI, you can control this behavior, where available for a connector, through its `--input-path` command option.
    * For the Unstructured Ingest Python library, you can control this behavior, where available for a connector, through its `<Prefix>IndexerConfig` class (where `<Prefix>` represents the connector provider's name, such as `Azure` for Azure.)
  </Step>

  <Step title="Post-Index Filter">
    After indexing, you might not want to download everything that was indexed.

    For example, you might want to download only files that match specific types, file names, paths, or sizes.

    For the Unstructured Ingest Python library, you can control this behavior through the `FiltererConfig` class.
  </Step>

  <Step title="Download">
    Using the information generated from the indexer and the filter, downloads the content as files on the local file system for processing. This may require manipulation of the data to prepare it for partitioning.

    For example, this could include information such as the path to a local directory to download files to.

    * For the Unstructured CLI, you can control this behavior through a connector's `--download-dir` command option.
    * For the Unstructured Ingest Python library, you can control this behavior through a connector's `<Prefix>DownloaderConfig` class.
  </Step>

  <Step title="Post-Download Filter">
    After downloading, if uncompression is enabled, you might not want to uncompress everything that was downloaded. The filter that was defined at the beginning is repeated here.
  </Step>

  <Step title="Uncompress">
    If enabled, searches for any compressed files (Unstructured supports TAR and ZIP) and uncompresses them.

    * For the Unstructured CLI, you can control this behavior through the `--uncompress` command option.
    * For the Unstructured Ingest Python library, you can control this behavior through the `UncompressConfig` class.
  </Step>

  <Step title="Post-Uncompress Filter">
    After downloading, and uncompressing if enabled, you might not want to partition everything that was downloaded, and uncompressed if enabled. The filter that was defined at the beginning is repeated here again.
  </Step>

  <Step title="Partition">
    Generates the structured enriched content from the local files that have been downloaded, uncompressed if enabled, and filtered. Both local-based partitioning and Unstructured-based partitioning is supported, with API services-based partitioning set to run asynchronously and local-based partitioning set to run through multiprocessing.

    For the Unstructured Ingest Python library, you can control this behavior through the `PartitionerConfig` class.
  </Step>

  <Step title="Chunk">
    Optionally, chunks the partitioned content. Chunking can also be run locally or through Unstructured, with asynchronous or multiprocessing behaviors set in the same approach as the partitioner.

    For the Unstructured Ingest Python library, you can control this behavior through the `ChunkerConfig` class.
  </Step>

  <Step title="Embed">
    Optionally, generates vector embeddings for each element in the structured output. Most of the time, this is done through API calls to a third-party embedding vendor and therefore runs asynchronously. But it can also use a locally available Hugging Face model, which will run through multiprocessing.

    For the Unstructured Ingest Python library, you can control this behavior through the `EmbedderConfig` class.
  </Step>

  <Step title="Stage">
    This is an optional step that does not apply most of the time. However, sometimes the data needs to be modified from the existing structure to better support the upload step, such as converting it to a CSV file for tabular-based destinations.

    For the Unstructured Ingest Python library, you can control this behavior, where available for a connector, through its `<Prefix>UploadStagerConfig` class.
  </Step>

  <Step title="Upload">
    Writes the local content to the destination. If no destination is provided, the local one will be used, which writes the final result to a location on the local filesystem. If batch uploads are needed, this will run in a single process with access to all documents. If batch uploads are not supported, all documents can be uploaded concurrently by using the asynchronous approach.

    For the Unstructured Ingest Python library, you can control this behavior through a connector's `<Prefix>UploaderConfig` class.
  </Step>
</Steps>

## Generate Python code examples

You can connect any available source connector to any available destination connector. However, the source connector code examples in the
documentation show connecting only to the local destination connector. Similarly, the destination connector code examples in the
documentation show connecting only to the local source connector.

To quickly generate an Unstructured Ingest Python library code example that connects *any* available source connector to *any* available destination connector,
do the following:

1. Open the [Unstructured Ingest Code Generator](https://huggingface.co/spaces/MariaK/unstructured-pipeline-builder) webpage.

2. Select your input (source) location type from the **Get unstructured documents from** drop-down list.

3. Select your output (destination) location type from the **Upload RAG-ready documents to** drop-down list.

4. Select your chunking strategy from the **Chunking strategy** drop-down list:

   * **None** - Do not chunk the data elements' content.
   * **basic** - Combine sequential data elements to maximally fill each chunk. However, do not mix `Table` and non-`Table` elements in the same chunk.
   * **by\_title** - Use the `basic` strategy and also preserve section boundaries. Optionally preserve page boundaries as well.
   * **by\_page** - Use the `basic` strategy and also preserve page boundaries.
   * **by\_similarity** - Use the `sentence-transformers/multi-qa-mpnet-base-dot-v1` embedding model to identify topically similar sequential elements and combine them into chunks. This strategy is availably only when calling Unstructured.

   To learn more, see [Chunking strategies](/api-reference/legacy-api/partition/chunking) and [Chunking configuration](/open-source/ingestion/ingest-configuration/chunking-configuration).

5. For any chunking strategy other than **None**:

   * Enter your chunk size in the **Chunk size (characters)** box, or leave the default of **1000** characters.
   * If you need to apply overlapping to the chunks, enter the chunk overlap size in the **Chunk overlap (characters)** box, or leave default of **20** characters.

   To learn more, see [Chunking configuration](/open-source/ingestion/ingest-configuration/chunking-configuration).

6. To generate vector embeddings, select the provider in the **Embedding provider** drop-down list.

   To learn more, see [Embedding configuraton](/open-source/ingestion/ingest-configuration/embedding-configuration).

7. Click **Generate code**.

8. Copy the example code from the **Generated Code** pane into your code project.

9. The code example will contain one or more environment variables that you must set for the code to run correctly. To learn what to
   set these variables to, click the documentation links that are below the **Generated Code** pane.

## Pricing

Calls to the Unstructured CLI or Unstructured Ingest Python library that are routed to Unstructured's software-as-a-service (SaaS)
for processing (for example, by specifying an
Unstructured API key and an Unstructured SaaS URL) require an Unstructured account for billing purposes.

Unstructured offers different account types with different pricing plans:

* <Icon icon="person" />  **Let's Go** and **Pay-As-You-Go** - A single user, with a single workspace, hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="building" />  **Business** - Multiple users and workspaces, with three options:

  * <Icon icon="people-group" />  **Business SaaS** - Hosted alongside other accounts on Unstructured's cloud infrastructure.
  * <Icon icon="shield-halved" />  **Dedicated instance** - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control.
  * <Icon icon="shield" />  **In-VPC** - Hosted within your own VPC on your own cloud infrastructure.

  **Business** accounts also allow for robust customization of Unstructured's features for your unique needs.

For more details, see the [Unstructured Pricing](https://unstructured.io/pricing) page.

To upgrade your account from **Let's Go** or **Pay-As-You-Go** to **Business**,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Some of these plans have billing details that are determined on a per-page basis.

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

## Learn more

* [Ingest configuration](/open-source/ingestion/ingest-configuration/overview) settings enable you to control how batches are sent and processed.
* [Source connectors](/open-source/ingestion/source-connectors/overview) enable you to send batches from local or remote locations to be ingested by Unstructured for processing.
* [Destination connectors](/open-source/ingestion/destination-connectors/overview) enable Unstructured to send the processed data to local or remote locations.

## See also

* The [Unstructured Pipelines](/pipelines/overview) enables you to send batches to Unstructured from remote locations, and to have Unstructured send the processed data to remote locations, all without using code or a CLI.


# Process files in batches by using the Unstructured Ingest Python library
Source: https://docs.unstructured.io/open-source/ingestion/python-ingest



The Unstructured Ingest Python library enables you to use Python code to send files in batches to Unstructured for processing, and to tell Unstructured where to deliver the processed data.

<Note>
  The Unstructured Ingest Python library does not work with the Unstructured API.

  For information about the Unstructured API, see the [Unstructured API Overview](/api-reference/workflow/overview).
</Note>

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured for processing:

<iframe title="YouTube video player" />

## Getting started

You can use the Unstructured Ingest Python library to process files locally, or you can use the Ingest Python library to send files in batches to Unstructured for processing.

Local processing does not use an Unstructured API key or API URL.

Using the Ingest Python library to send files in batches to Unstructured for processing is more robust, and usage is billed to you on a pay-as-you-go basis. Usage requires an Unstructured API key and API URL, as follows:

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/api-reference/overview#pricing).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Get your Unstructured API key:<br />

   a. After you sign in to your Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business** account, click **API Keys** on the sidebar.<br />

   <Note>
     For a **Business** account, before you click **API Keys**, make sure you have selected the organizational workspace you want to create an API key
     for. Each API key works with one and only one organizational workspace. [Learn more](/pipelines/account/workspaces#create-an-api-key-for-a-workspace).
   </Note>

   b. Click **Generate API Key**.<br />
   c. Follow the on-screen instructions to finish generating the key.<br />
   d. Click the **Copy** icon next to your new key to add the key to your system's clipboard. If you lose this key, simply return and click the **Copy** icon again.<br />

3) The Unstructured API URL for Unstructured Ingest was provided to you when your Unstructured account was created.
   If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Note>
  The default URL for Unstructured Ingest is the same as the default URL for the legacy Unstructured Partition Endpoint: `https://api.unstructuredapp.io/general/v0/general`.
  However, you should always use the URL that was provided to you when your Unstructured account was created.
</Note>

## Installation

One approach to get started quickly with the Unstructured Ingest Python library is to install Python and then run the following command:

```bash theme={null}
pip install unstructured-ingest
```

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

For additional installation options, see the [Unstructured Ingest Python library](/open-source/ingestion/python-ingest) in the [Ingest](/open-source/ingestion/overview) section.

<Info>To migrate from older, deprecated versions of the Ingest Python library that used `pip install unstructured`, see the [migration guide](/open-source/ingestion/overview#migration-guide).</Info>

## Usage

For example, to use the Unstructured Ingest Python library to ingest files from a local source (input) location and to deliver the processed data to an Azure Storage account destination (output) location:

<CodeGroup>
  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.azure import (
      AzureConnectionConfig,
      AzureAccessConfig,
      AzureUploaderConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          uploader_config=AzureUploaderConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL"))
      ).run()
  ```
</CodeGroup>

To learn how to use the Unstructured Ingest Python library to work with a specific source (input) and destination (output) location, see the Python code examples for the [source](/open-source/ingestion/source-connectors/overview) and [destination](/open-source/ingestion/destination-connectors/overview) connectors that are available for you to choose from.

See also the [ingest configuration](/open-source/ingestion/ingest-configuration/overview) settings that enable you to further control how batches are sent and processed.

## Migration guide

The Unstructured Ingest Python library introduced breaking changes in version 0.7.0 as follows:

* The library has moved the calling pattern (formerly commonly referred to as "v2") up one level in the package. If you are using this calling pattern
  in your existing code, beginning with version 0.7.0 you must remove `.v2` from your related `from...import` statements. For example, if you previously had statments
  such as these:

  ```python theme={null}
  from unstructured_ingest.v2.pipeline.pipeline import Pipeline
  from unstructured_ingest.v2.interfaces import ProcessorConfig
  ```

  Beginning with version 0.7.0, you must change to these by removing `.v2`:

  ```python theme={null}
  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  ```

  If you do not make these changes beginning with version 0.7.0, your existing code will no longer work. To run your existing code
  without making these changes, you must pin your installed library version to 0.6.4 or earlier.

* The earlier calling pattern (commonly referred to as "v1") has been deprecated and removed from the library and will no longer work beginning with version 0.7.0. If you are using this earlier "v1" calling pattern in your code, you must
  either manually migrate your code to the new calling pattern (formerly commonly referred to as "v2"), or pin your installed library version to 0.6.4 or earlier to keep using this earlier "v1" calling pattern.

  To learn how to use the new calling pattern (formerly commonly referred to as "v2"), see the code examples in the
  [source connectors](/open-source/ingestion/source-connectors/overview) and [destination connectors](/open-source/ingestion/destination-connectors/overview) documentation.


# Airtable
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/airtable



Connect Airtable to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your data and store structured outputs locally on your filesystem.

The requirements are as follows.

* An [Airtable](https://www.airtable.com/) account. [Create a free Airtable account](https://airtable.com/signup).

* An Airtable personal access token. [Create a personal access token](https://support.airtable.com/docs/creating-and-using-api-keys-and-access-tokens).

  <iframe title="YouTube video player" />

* The ID of the Airtable base to access. [Create a base](https://www.airtable.com/guides/build/create-a-base). [Get a base's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

  <iframe title="YouTube video player" />

* The ID of the table to access in the base. [Create a table](https://www.airtable.com/guides/build/create-a-table). [Get a table's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

* The ID of the view to access in the table. [Create a view](https://www.airtable.com/guides/build/create-custom-views-of-data). [Get a view's ID](https://support.airtable.com/docs/finding-airtable-ids#finding-base-url-ids).

By default, Unstructured processes all tables from all bases within an Airtable organization. You can limit the
tables that Unstructured ingests data from within Airtable by specifying a list of Airtable paths.
An Airtable path uses the following structure: `base_id/table_id(optional)/view_id(optional)`

For example, given the following example Airtable URL:

```text theme={null}
https://airtable.com/appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit/viwDcpzf9dP0Gqz5J
```

* The base's ID is `appr9nKeXLAtg6bgn`. The base's path is `appr9nKeXLAtg6bgn`.
* The table's ID is `tblZ8uT1GY7NLbWit`. The table's path is `appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit`.
* The view's ID is `viwDcpzf9dP0Gqz5J`. The view's path is `appr9nKeXLAtg6bgn/tblZ8uT1GY7NLbWit/viwDcpzf9dP0Gqz5J`.

You can call the Airtable API to get lists of available IDs for Airtable bases, tables, and views in bulk, as follows:

* [Base IDs](https://airtable.com/developers/web/api/list-bases)
* [Table and view IDs](https://airtable.com/developers/web/api/get-base-schema)
* [Base, table, and view IDs](https://pyairtable.readthedocs.io/en/latest/metadata.html)

The Airtable connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[airtable]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `AIRTABLE_TOKEN` - The Airtable personal access token, represented by `--personal-access-token` (CLI) or `personal_access_token` (Python).
* `AIRTABLE_PATHS` - The list of Airtable paths to process, represented by `--list-of-paths` (CLI) or `list_of_paths` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    airtable \
      --personal-access-token $AIRTABLE_PERSONAL_ACCESS_TOKEN \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --reprocess \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.airtable import (
      AirtableIndexerConfig,
      AirtableDownloaderConfig,
      AirtableConnectionConfig,
      AirtableAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig

  # Chunking is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AirtableIndexerConfig(list_of_paths=[os.getenv("AIRTABLE_PATHS")]),
          downloader_config=AirtableDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=AirtableConnectionConfig(
              access_config=AirtableAccessConfig(
                  personal_access_token=os.getenv("AIRTABLE_TOKEN")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Astra DB
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/astradb



Connect Astra DB to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* An IBM Cloud account or DataStax account.

  * For an IBM Cloud account, [sign up](https://cloud.ibm.com/registration) for an IBMid, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your IBMid.
  * For a DataStax account, [sign up](https://astra.datastax.com/signup) for a DataStax account, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your DataStax account.

* An Astra DB database in the DataStax account. To create a database:

  a. After you sign in to DataStax, click **Create database**.<br />
  b. Click the **Serverless (vector)** tile, if it is not already selected.<br />
  c. For **Database name**, enter some unique name for the database.<br />
  d. Select a **Provider** and a **Region**, and then click **Create database**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).

* An application token for the database. To create an application token:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Overview** tab, under **Database Details**, in the **Application Tokens** tile, click **Generate Token**.<br />
  c. Enter some **Token description** and select and **Expiration** time period, and then click **Generate token**.<br />
  d. Save the application token that is displayed to a secure location, and then click **Close**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).

* A keyspace in the database. To create a keyspace:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select **Create keyspace**.<br />
  c. Enter some **Keyspace name**, and then click **Add keyspace**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-keyspaces.html#keyspaces).

* A collection in the keyspace.

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * An existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. If Unstructured generates embeddings,
      the new collection's name will be `u<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `u<short-workflow-id`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * For the source connector only, an existing collection is required.
  * For the destination connector only, an existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. The new collection's name will be `unstructuredautocreated`.

  To create a collection yourself:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select the name of the target keyspace.<br />
  c. In the **Collections** list, select **Create collection**.<br />
  d. Enter some **Collection name**.<br />
  e. Turn on **Vector-enabled collection**, if it is not already turned on.<br />
  f. Choose a mode for **Embedding generation method**. See [Astra DB generated embeddings](#astra-db-generated-embeddings).<br />
  g. If you chose **Bring my own**, enter the number of dimensions for the embedding model that you plan to use.<br />
  h. For **Similarity metric**, select **Cosine**.<br />
  i. Click **Create collection**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

The Astra DB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[astradb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `ASTRA_DB_API_ENDPOINT` - The API endpoint for the Astra DB database, represented by `--api-endpoint` (CLI) or `api_endpoint` (Python). To get the endpoint, see the **Database Details > API Endpoint** value on your database's **Overview** tab.
* `ASTRA_DB_APPLICATION_TOKEN` - The database application token value for the database, represented by `--token` (CLI) or `token` (Python). To get the token, see the **Database Details > Application Tokens** box on your database's **Overview** tab.
* `ASTRA_DB_KEYSPACE` - The name of the keyspace for the database, represented by `--keyspace` (CLI) or `keyspace` (Python).
* `ASTRA_DB_COLLECTION` - The name of the collection for the keyspace, represented by `--collection-name` (CLI) or `collection_name` (Python). If no value is provided, see the beginning of this article for the behavior at run time.

Additional settings include:

* For the source connector only, `--fields` (CLI) or `fields` (Python): Optionally, a comma-separated list (CLI) or an array of strings (Python) of fields
  to include in the output. The default is to include all fields, if not otherwise specified.
* For the destination connector only, `--flatten-metadata` (CLI) or `flatten_metadata=True` (Python): Optionally, whether to "flatten" the metadata. Specifically, the metadata key values are
  brought to the top level of the element, and the `metadata` key itself is removed. To not flatten the metadata (the default), specify `--no-flatten_metadata` (CLI) or
  `flatten_metadata=False` (Python). The default is is to not flatten the metadata if not otherwise specified.
* For the destination connector only, `--binary-encode-vectors` (CLI) or `binary_encode_vectors=True` (Python): Optionally, control when to upload vectors in a binary format. By default, vectors are binary encoded (`true`), which is more efficient. To upload vectors as a human-readable list of floats instead, specify `--no-binary-encode-vectors` (CLI) or `binary_encode_vectors=False` (Python). **WARNING**: Disabling this option may make the upload slower!
* For the destination connector only, `--enable-lexical-search` (CLI) or `enable_lexical_search=True` (Python): Optionally, enable lexical and hybrid search.
* For the destination connector only, `--astra-generated-embeddings` (CLI) or `astra_generated_embeddings=True` (Python): Optionally, use an Astra vectorize integration for embeddings.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    astradb \
      --api-endpoint $ASTRA_DB_API_ENDPOINT \
      --token $ASTRA_DB_APPLICATION_TOKEN \
      --collection-name $ASTRA_DB_COLLECTION \
      --keyspace $ASTRA_DB_KEYSPACE \
      --fields record_id,content \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.astradb import (
      AstraDBIndexerConfig,
      AstraDBDownloaderConfig,
      AstraDBConnectionConfig,
      AstraDBAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AstraDBIndexerConfig(
              collection_name=os.getenv("ASTRA_DB_COLLECTION"),
              keyspace=os.getenv("ASTRA_DB_KEYSPACE"),
              batch_size=20
          ),
          downloader_config=AstraDBDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              fields=["record_id", "content"]
          ),
          source_connection_config=AstraDBConnectionConfig(
              access_config=AstraDBAccessConfig(
                  api_endpoint=os.getenv("ASTRA_DB_API_ENDPOINT"),
                  token=os.getenv("ASTRA_DB_APPLICATION_TOKEN")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Azure
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/azure



Connect Azure Storage to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Azure Storage account requirements:

<iframe title="YouTube video player" />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  * **Read** and **List** for reading from the container only.
  * **Write** and **List** for writing to the container only.
  * **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these requirements:

* An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

  <iframe title="YouTube video player" />

* An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

  <iframe title="YouTube video player" />

  <iframe title="YouTube video player" />

* The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

  Create an SAS token (recommended):

  <iframe title="YouTube video player" />

  Get an access key or connection string:

  <iframe title="YouTube video player" />

The Azure Storage account connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[azure]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `AZURE_STORAGE_REMOTE_URL` - The Azure Storage remote URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

  The remote URL takes the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* `AZURE_STORAGE_ACCOUNT_NAME` - The name of the Azure Storage account, represented by `--account-name` (CLI) or `account_name` (Python).

* `AZURE_STORAGE_ACCOUNT_KEY`, `AZURE_STORAGE_CONNECTION_STRING`, or `AZURE_STORAGE_SAS_TOKEN` - The name of the key, connection string, or SAS token for the Azure Storage account, depending on your security configuration, represented by `--account-key` (CLI) or `account_key` (Python), `--connection-string` (CLI) or `connection_string` (Python), and `--sas_token` (CLI) or `sas_token` (Python), respectively.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    azure \
      --remote-url $AZURE_STORAGE_REMOTE_URL \
      --account-name $AZURE_STORAGE_ACCOUNT_NAME \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.azure import (
      AzureIndexerConfig,
      AzureDownloaderConfig,
      AzureConnectionConfig,
      AzureAccessConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=AzureIndexerConfig(remote_url=os.getenv("AZURE_STORAGE_REMOTE_URL")),
          downloader_config=AzureDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=AzureConnectionConfig(
              access_config=AzureAccessConfig(
                  account_name=os.getenv("AZURE_STORAGE_ACCOUNT_NAME"),
                  account_key=os.getenv("AZURE_STORAGE_ACCOUNT_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Box
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/box



Connect Box to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

1. Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2. A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

   * In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
   * In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
   * Click **Save Changes** before continuing.

4. Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5. Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

   * In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
   * **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
   * Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6. The private key configuration JSON file for the Box Custom App, or a string that contains this file's contents.

   * To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location.
   * To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as
     a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself).
     To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
     In this command, replace `<path-to-downloaded-key-file>` with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.

     * For macOS or Linux:

       ```text theme={null}
       tr -d '\n' < <path-to-downloaded-key-file>
       ```

     * For Windows:

       ```text theme={null}
       (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
       ```

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

Permissions are defined in Box through *collaborations*, which function similar to access control lists. Collaboration objects are attached to folders, and those collaborations cascade down to everything inside that folder. A file's effective permissions come from the chain of ancestor folders above it. For more information, see [Collaborations overview](https://developer.box.com/guides/collaborations/) in the Box developer documentation.

To determine a file's effective permissions, Unstructured traverses the full folder hierarchy for each file, collecting collaborations at every level, and merging them. It also includes collaborations applied directly to the file itself. For more information, see [List folder collaborations](https://developer.box.com/reference/get-folders-id-collaborations/)  and [List file collaborations](https://developer.box.com/reference/get-files-id-collaborations/) in the Box developer documentation.

When Unstructured compiles a file's effective permissions:

* If the same user or group appears at multiple levels with different roles, Unstructured chooses the least-restrictive role.
* Unstructured does not include the following in the permission metadata. No sentinel value is written into the permission metadata.
  * `all_users_group`: Box's built-in group representing every user in the enterprise.
  * `is_access_only`: Collaboration property that determines whether, for collaborators, to display the items in the All Files list and let them see the path to the root folder for the shared item.
* Unstructured only includes collaborations with a status of `accepted`. It does not include collaborations with a status of `pending` or `rejected`.

Unstructured includes a maximum of 500 permissions per file.

Unstructured maps Box roles to `read`, `update`, and `delete` access:

| Box role             | Box access                                               | Unstructured permission metadata |
| -------------------- | -------------------------------------------------------- | -------------------------------- |
| `owner`              | Full access                                              | `read`, `update`, `delete`       |
| `co-owner`           | Full access except transfer ownership                    | `read`, `update`, `delete`       |
| `editor`             | View, download, upload, edit, delete, copy, move, rename | `read`, `update`                 |
| `viewer`             | Preview, download, comment                               | `read`                           |
| `previewer`          | Preview                                                  | `read`                           |
| `viewer uploader`    | Preview, download, upload                                | `read`                           |
| `previewer uploader` | Preview, upload                                          | `read`                           |
| `uploader`           | Upload and see item names                                | excluded entirely                |

Note the following:

* Unstructured excludes the `uploader` role because it confers no `read`, `update`, or `delete` access.
* Unstructured maps the Box `editor` role to `read` and `update` access but not `delete`, even though that role does allow deleting files.

For more information, see [Understanding collaborator permission level](https://support.box.com/hc/en-us/articles/360044196413-Understanding-Collaborator-Permission-Levels) at Box Support.

### Identifier format

Users and groups are identified by the numeric ID that Box assigns them. To retrieve information about a specific user or group, use appropriate Box API with the corresponding ID:

* [Get user](https://developer.box.com/reference/get-users-id/)
* [Get group](https://developer.box.com/reference/get-groups-id/)

### Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["11446688", "11336699"],
                            "groups": ["11223344"]
                        }
                    },
                    {
                        "update": {
                            "users": ["11446688"],
                            "groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

The Box connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[box]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `BOX_APP_CONFIG` - The local path to the downloaded private key configuration JSON file for the Box Custom App,
  or a single-line string that contains the contents of this file, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
* `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).
  This URL must take the format `box://<path/to/folder/in/account>`.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    box \
      --box-app-config $BOX_APP_CONFIG \
      --remote-url $BOX_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.box import (
      BoxAccessConfig,
      BoxConnectionConfig,
      BoxIndexerConfig,
      BoxDownloaderConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=BoxIndexerConfig(remote_url=os.getenv("BOX_REMOTE_URL")),
          downloader_config=BoxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=BoxConnectionConfig(
              access_config=BoxAccessConfig(
                  box_app_config=os.getenv("BOX_APP_CONFIG")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Confluence
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/confluence



Connect Confluence to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your spaces and pages and store structured outputs locally on your filesystem.

The requirements are as follows.

* A [Confluence Cloud account](https://www.atlassian.com/software/confluence/pricing) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html).

* The site URL for your [Confluence Cloud account](https://community.atlassian.com/t5/Confluence-questions/confluence-cloud-url/qaq-p/1157148) or
  [Confluence Data Center installation](https://confluence.atlassian.com/confkb/how-to-find-your-site-url-to-set-up-the-confluence-data-center-and-server-mobile-app-938025792.html).

* A user in your [Confluence Cloud account](https://confluence.atlassian.com/cloud/invite-edit-and-remove-users-744721624.html) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html).

* The user must have the correct permissions in your
  [Conflunce Cloud account](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/permissions-and-restrictions-139557.html) to
  access the target spaces and pages.

* One of the following:

  * For Confluence Cloud or Confluence Data Center, the target user's name or email address, and password.
    [Change a Confluence Cloud user's password](https://support.atlassian.com/confluence-cloud/docs/change-your-confluence-password/).
    [Change a Confluence Data Center user's password](https://confluence.atlassian.com/doc/change-your-password-139416.html).
  * For Confluence Cloud only, the target user's name or email address, and API token.
    [Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
  * For Confluence Data Center only, the target user's personal access token (PAT).
    [Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).

* Optionally, the keys (not display names) of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access. To get a space's key,
  which is different from a space's display name, open the space in your web browser and look at the URL. The space's key is the part of the URL after `spaces/` but before the next `/` after that.

The following video provides related setup information for Confluence Cloud:

<iframe title="YouTube video player" />

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "11111:11111111-1111-1111-1111-111111111111"
                            ],
                            "groups": [
                                "22222222-2222-2222-2222-22222222",
                                "33333333-3333-3333-3333-33333333"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "44444:44444444-4444-4444-4444-44444444",
                                "55555:55555555-5555-5555-5555-55555555"
                            ],
                            "groups": [
                                "66666666-6666-6666-6666-66666666",
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "77777:77777777-7777-7777-7777-77777777"
                            ],
                            "groups": [
                                "88888888-8888-8888-8888-88888888"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Confluence user, use the user's ID (also known as their *account ID*) from the preceding output to call the
[GET /wiki/rest/api/user](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-users/#api-wiki-rest-api-user-get)
operation in the Confluence REST API.

To look up information about a particular Confluence group, use the group's ID from the preceding output to call the
[GET /wiki/rest/api/group/by-id](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-group/#api-wiki-rest-api-group-by-id-get)
operation in the Confluence REST API.

The Confluence connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[confluence]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `CONFLUENCE_URL` - The target Confluence site's URL, represented by `--url` (CLI) or `url` (Python).
* One of the following:

  * For API token authentication: `CONFLUENCE_USERNAME` and `CONFLUENCE_API_TOKEN` - The name or email address, and API token of the target Confluence user, represented by `--username` (CLI) or `username` (Python) and `--api-token` (CLI) or `api-token` (Python), respectively.
  * For personal access token (PAT) authentication: `CONFLUENCE_PERSONAL_ACCESS_TOKEN` - The PAT for the target Confluence user, represented by `--token` (CLI) or `token` (Python).
  * For password authentication: `CONFLUENCE_USERNAME` and `CONFLUENCE_PASSWORD` - The name or email address, and password of the target Confluence user, represented by `--username` (CLI) or `username` (Python) and `--password` (CLI) or `password` (Python), respectively.

Additional settings include:

* `--spaces` (CLI) or `spaces` (Python): Optionally, the list of the keys (not display names) of the specific spaces to access, expressed as a comma-separated list of strings (CLI) or an array of strings (Python), with each string representing a space's key (not display name). The default is no specific spaces, if not otherwise specified.
* `--max-num-of-spaces` (CLI) or `max_num_of_spaces` (Python): Optionally, the maximum number of spaces to access, expressed as an integer. The default value is `500` if not otherwise specified.
* `--max-num-of-docs-from-each-space` (CLI) or `max_num_of_docs_from_each_space` (Python): Optionally, the maximum number of documents to access from each space, expressed as an integer. The default value is `100` if not otherwise specified.
* `--cloud` or `--no-cloud` (CLI) or `cloud` (Python): Optionally, whether to use Confluence Cloud (`--cloud` for CLI or `cloud=True` for Python). The default is `--no-cloud` (CLI) or `cloud=False` (Python) if not otherwise specified.
* `--extract-images` (CLI) or `extract_images` (Python): Optionally, download images and replace the HTML content with Base64-encoded images. The default is `--no-extract-images` (CLI) or `extract_images=False` (Python) if not otherwise specified.
* `--extract-files` (CLI) or `extract_files` (Python): Optionally, download any embedded files. The default is `--no-extract-files` (CLI) or `extract_files=False` (Python) if not otherwise specified.
* `--force-download` (CLI) or `force_download` (Python): Optionally, re-download extracted files even if they already exist locally. The default is `--no-force-download` (CLI) or `force_download=False` (Python) if not otherwise specified.
* `--allow-list` (CLI) or `allow_list`: Optionally, a comma-separated list (CLI) or an array of strings (Python) of allowed URLs to download. By default, the base URL that the original HTML came from is used, if not otherwise specified.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # For API token authentication:
  unstructured-ingest \
    confluence \
      --api-token $CONFLUENCE_API_TOKEN \
      --url $CONFLUENCE_URL \
      --username $CONFLUENCE_USERNAME \
      --cloud \
      --spaces <space-key>,<space-key> \
      --max-num-of-spaces 500 \
      --max-num-of-docs-from-each-space 150 \
      --extract-images \
      --extract-files \
      --force-download \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface

  # For personal access token (PAT) authentication:
  unstructured-ingest \
    confluence \
      --token $CONFLUENCE_PERSONAL_ACCESS_TOKEN \
      --url $CONFLUENCE_URL \
      --spaces <space-key>,<space-key> \
      --max-num-of-spaces 500 \
      --max-num-of-docs-from-each-space 150 \
      --extract-images \
      --extract-files \
      --force-download \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface

  # For password authentication:
  unstructured-ingest \
    confluence \
      --password $CONFLUENCE_PASSWORD \
      --url $CONFLUENCE_URL \
      --username $CONFLUENCE_USERNAME \
      --no-cloud \
      --spaces luke,paul \
      --max-num-of-spaces 500 \
      --max-num-of-docs-from-each-space 150 \
      --extract-images \
      --extract-files \
      --force-download \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.confluence import (
      ConfluenceIndexerConfig,
      ConfluenceDownloaderConfig,
      ConfluenceConnectionConfig,
      ConfluenceAccessConfig
  )

  # Chunking and embedding are optional.

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ConfluenceIndexerConfig(
              spaces=["<space-key>", "<space-key>"],
              max_num_of_spaces=500,
              max_num_of_docs_from_each_space=150
          ),
          downloader_config=ConfluenceDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
              extract_images=True,
              extract_files=True,
              force_download=True,
              allow_list=[] 
          ),
          # For API token authentication:
          source_connection_config=ConfluenceConnectionConfig(
              access_config=ConfluenceAccessConfig(
                  api_token=os.getenv("CONFLUENCE_API_TOKEN")
              ),
              url=os.getenv("CONFLUENCE_URL"),
              username=os.getenv("CONFLUENCE_USERNAME"), 
              cloud=True
          ),

          # For personal access token (PAT) authentication:
          # source_connection_config=ConfluenceConnectionConfig(
          #     access_config=ConfluenceAccessConfig(
          #         token=os.getenv("CONFLUENCE_PERSONAL_ACCESS_TOKEN")
          #     ),
          #     url=os.getenv("CONFLUENCE_URL")
          # ),

          # For password authentication:
          # source_connection_config=ConfluenceConnectionConfig(
          #     access_config=ConfluenceAccessConfig(
          #         password=os.getenv("CONFLUENCE_PASSWORD")                
          #     ),
          #     url=os.getenv("CONFLUENCE_URL"),
          #     username=os.getenv("CONFLUENCE_USERNAME"), 
          #     cloud=False
          # ),

          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Couchbase
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/couchbase



Connect Couchbase Database to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Couchbase Capella clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe title="YouTube video player" />

For Couchbase Capella, you will need:

* A [Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier).
* A [Couchbase Capella cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started).
* A [bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket),
  [scope](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-scope),
  and [collection](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-collection)
  on the cluster.
* The cluster's [public connection string](https://docs.couchbase.com/cloud/get-started/connect.html#connect-from-sdk-cbsh-cli-or-ide).
* The [cluster access name (username) and secret (password)](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials).
* [Incoming IP address allowance](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) for the cluster.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

For a local Couchbase server, you will need:

* [Installation of a local Couchbase server](https://docs.couchbase.com/server/current/getting-started/start-here.html).
* [Connection details](https://docs.couchbase.com/server/current/guides/connect.html) to the local Couchbase server.

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

The Couchbase DB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[couchbase]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables are required for the Couchbase Connector:

* `CB_CONN_STR` - The Connection String for the Couchbase server, represented by `--connection-string` (CLI) or `connection_string` (Python).
* `CB_USERNAME` - The username for the Couchbase server, represented by `--username` (CLI) or `username` (Python).
* `CB_PASSWORD` - The password for the Couchbase server, represented by `--password` (CLI) or `password` (Python).
* `CB_BUCKET` - The name of the bucket in the Couchbase server, represented by `--bucket` (CLI) or `bucket` (Python).
* `CB_SCOPE` - The name of the scope in the bucket, represented by `--scope` (CLI) or `scope` (Python).
* `CB_COLLECTION` - The name of the collection in the scope, represented by `--collection` (CLI) or `collection` (Python).

Additional available settings include:

* `--collection-id` (CLI) or `collection_id` in `CouchbaseDownloaderConfig` (Python) - Optional for the source connector. The\
  unique key of the ID field in the collection. The default is `id` if not otherwise specified.
  [Learn more](https://docs.couchbase.com/server/current/learn/services-and-indexes/indexes/indexing-and-query-perf.html#introduction-document-keys).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    couchbase \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --connection-string "$COUCHBASE_CONNECTION_STRING" \
      --username $COUCHBASE_USERNAM \
      --password $COUCHBASE_PASSWORD \
      --bucket $COUCHBASE_BUCKET \
      --scope $COUCHBASE_SCOPE \
      --collection $COUCHBASE_COLLECTION \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.couchbase import (
      CouchbaseIndexerConfig,
      CouchbaseDownloaderConfig,
      CouchbaseConnectionConfig,
      CouchbaseAccessConfig,
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=CouchbaseIndexerConfig(batch_size=100),
          downloader_config=CouchbaseDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=CouchbaseConnectionConfig(
              access_config=CouchbaseAccessConfig(
                  password=os.getenv("CB_PASSWORD")
              ),
              username=os.getenv("CB_USERNAME"),
              connection_string=os.getenv("CB_CONN_STR"),
              bucket=os.getenv("CB_BUCKET"),
              scope=os.getenv("CB_SCOPE"),
              collection=os.getenv("CB_COLLECTION")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Databricks Volumes
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/databricks-volumes



Connect Databricks Volumes to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A Databricks account on [AWS](https://docs.databricks.com/getting-started/free-trial.html),
  [Azure](https://learn.microsoft.com/azure/databricks/getting-started/), or
  [GCP](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or
  [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* The workspace's URL. Get the workspace URL for
  [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
  [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
  or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

  Examples:

  * AWS: `https://<workspace-id>.cloud.databricks.com`
  * Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
  * GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

  <Note>
    Do not add a trailing slash (`/`) to the workspace URL.
  </Note>

* The Databricks authentication details. For more information, see the documentation for
  [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
  or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

  For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), the following Databricks authentication types are supported:

  * Databricks OAuth machine-to-machine (M2M) authentication for\
    [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

    You will need the **Client ID** (or **UUID** or **Application** ID) and OAuth **Secret** (client secret) values for the corresponding service principal.
    Note that for Azure, only Databricks managed service principals are supported. Microsoft Entra ID managed service principals are not supported.

    The following video shows how to create a Databricks managed service principal:

    <iframe title="YouTube video player" />

  * Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html).

    You will need the personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  For [Unstructured Ingest](/open-source/ingestion/overview), the following Databricks authentication types are supported:

  * For Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html): The personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  * For username and password (basic) authentication ([AWS](https://docs.databricks.com/archive/dev-tools/basic.html) only): The user's name and password values.

  * For OAuth machine-to-machine (M2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html)): The client ID and OAuth secret values for the corresponding service principal.

  * For OAuth user-to-machine (U2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-u2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-u2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-u2m.html)): No additional values.

  * For Azure managed identities (formerly Managed Service Identities (MSI) authentication) ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-mi) only): The client ID value for the corresponding managed identity.

  * For Microsoft Entra ID service principal authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-sp) only): The tenant ID, client ID, and client secret values for the corresponding service principal.

  * For Azure CLI authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-cli) only): No additional values.

  * For Microsoft Entra ID user authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/user-aad-token) only): The Entra ID token for the corresponding Entra ID user.

  * For Google Cloud Platform credentials authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-creds.html) only): The local path to the corresponding Google Cloud service account's credentials file.

  * For Google Cloud Platform ID authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-id.html) only): The Google Cloud service account's email address.

* The name of the parent catalog in Unity Catalog for
  [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
  [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
  [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html) for the volume.

* The name of the parent schema (formerly known as a database) in Unity Catalog for
  [AWS](https://docs.databricks.com/schemas/create-schema.html),
  [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
  [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) for the volume.

* The name of the volume in Unity Catalog for [AWS](https://docs.databricks.com/tables/managed.html),
  [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
  [GCP](https://docs.gcp.databricks.com/tables/managed.html), and optionally any path in that volume that you want to access directly, beginning with the volume's root.

* The Databricks workspace user or service principal must have the following *minimum* set of privileges to read from or write to the
  existing volume in Unity Catalog:

  * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
  * `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
  * `READ VOLUME` and `WRITE VOLUME` on the volume.

  The following videos show how to create and set privileges for a catalog, schema (formerly known as a database), and volume in Unity Catalog.

  <iframe title="YouTube video player" />

  Learn more about how to check and set Unity Catalog privileges for
  [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

The Databricks Volumes connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[databricks-volumes]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DATABRICKS_HOST` - The Databricks host URL, represented by `--host` (CLI) or `host` (Python).

  <Note>
    Do not add a trailing slash (`/`) to the host URL.
  </Note>

* `DATABRICKS_CATALOG` - The Databricks catalog name for the Volume, represented by `--catalog` (CLI) or `catalog` (Python).

* `DATABRICKS_SCHEMA` - The Databricks schema name for the Volume, represented by `--schema` (CLI) or `schema` (Python). If not specified, `default` is used.

* `DATABRICKS_VOLUME` - The Databricks Volume name, represented by `--volume` (CLI) or `volume` (Python).

* `DATABRICKS_VOLUME_PATH` - Any optional path to access within the volume, specified by `--volume-path` (CLI) or `volume_path` (Python).

Environment variables based on your authentication type, depending on your cloud provider:

* For Databricks personal access token authentication (AWS, Azure, and GCP):

  * `DATABRICKS_TOKEN` - The personal access token, represented by `--token` (CLI) or `token` (Python).

* For username and password (basic) authentication (AWS only): The user's name and password values.

  * `DATABRICKS_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
  * `DATABRICKS_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

* For OAuth machine-to-machine (M2M) authentication (AWS, Azure, and GCP): The client ID and OAuth secret values for the corresponding service principal.

  * `DATABRICKS_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--client-id` (CLI) or `client_id` (Python).
  * `DATABRICKS_CLIENT_SECRET` - The client ID and OAuth secret values for the corresponding service principal, represented by `--client-secret` (CLI) or `client_secret` (Python).

* For OAuth user-to-machine (U2M) authentication (AWS, Azure, and GCP): No additional environment variables.

* For Azure managed identities (MSI) authentication (Azure only):

  * `ARM_CLIENT_ID` - The client ID value for the corresponding managed identity, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
  * If the target identity has not already been added to the workspace, then you must also specify the
    `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

* For Microsoft Entra ID service principal authentication (Azure only):

  * `ARM_TENANT_ID` - The tenant ID value for the corresponding service principal, represented by `--azure-tenant-id` (CLI) or `azure_tenant_id` (Python).
  * `ARM_CLIENT_ID` - The client ID value for the corresponding service principal, represented by `--azure-client-id` (CLI) or `azure_client_id` (Python).
  * `ARM_CLIENT_SECRET` - The client secret value for the corresponding service principal, represented by `--azure-client-secret` (CLI) or `azure_client_secret` (Python).
  * If the service principal has not already been added to the workspace, then you must also specify the
    `DATABRICKS_AZURE_RESOURCE_ID`, represented by `--azure-workspace-resource-id` (CLI) or `azure_workspace_resource_id` (Python).

* For Azure CLI authentication (Azure only): No additional environment variables.

* For Microsoft Entra ID user authentication (Azure only):

  * `DATABRICKS_TOKEN` - The Entra ID token for the corresponding Entra ID user, represented by `--token` (CLI) or `token` (Python).

* For Google Cloud Platform credentials authentication (GCP only):

  * `GOOGLE_CREDENTIALS` - The local path to the corresponding Google Cloud service account's credentials file, represented by `--google-credentials` (CLI) or `google_credentials`

* For Google Cloud Platform ID authentication (GCP only):

  * `GOOGLE_SERVICE_ACCOUNT` - The Google Cloud service account's email address, represented by `--google-service-account` (CLI) or `google_service_account` (Python).

* Alternatively, you can store the preceding settings in a local
  [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth/config-profiles.html) and then just
  refer to the profile's name:

  * `DATABRICKS_PROFILE` - The name of the Databricks configuration profile, represented by `--profile` (CLI) or `profile` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    databricks-volumes \
      --profile $DATABRICKS_PROFILE \
      --host $DATABRICKS_HOST \
      --catalog $DATABRICKS_CATALOG \
      --schema $DATABRICKS_SCHEMA \
      --volume $DATABRICKS_VOLUME \
      --volume-path $DATABRICKS_VOLUME_PATH \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  # For all supported Databricks authentication types, you can import this:
  from unstructured_ingest.processes.connectors.databricks.volumes import (
      DatabricksVolumesIndexerConfig,
      DatabricksVolumesConnectionConfig,
      DatabricksVolumesAccessConfig,
      DatabricksVolumesDownloaderConfig
  )

  # Alternatively, for supported Databricks on AWS authentication types only, you can import this:
  # from unstructured_ingest.processes.connectors.databricks.volumes_aws import (
  #     DatabricksAWSVolumesIndexerConfig,
  #     DatabricksAWSVolumesConnectionConfig,
  #     DatabricksAWSVolumesAccessConfig,
  #     DatabricksAWSVolumesDownloaderConfig
  # )

  # Alternatively, for supported Azure Databricks authentication types only, you can import this:
  # from unstructured_ingest.processes.connectors.databricks.volumes_azure import (
  #     DatabricksAzureVolumesIndexerConfig,
  #     DatabricksAzureVolumesConnectionConfig,
  #     DatabricksAzureVolumesAccessConfig,
  #     DatabricksAzureVolumesDownloaderConfig
  # )

  # Alternatively, for supported Databricks on Google Cloud authentication types only, you can import this:
  # from unstructured_ingest.processes.connectors.databricks.volumes_gcp import (
  #     DatabricksGoogleVolumesIndexerConfig,
  #     DatabricksGoogleVolumesConnectionConfig,
  #     DatabricksGoogleVolumesAccessConfig,
  #     DatabricksGoogleVolumesDownloaderConfig
  # )

  from unstructured_ingest.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(reprocess=True),
          indexer_config=DatabricksVolumesIndexerConfig(recursive=True),
          # For specifying a Databricks configuration profile:
          downloader_config=DatabricksVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DatabricksVolumesConnectionConfig(
              access_config=DatabricksVolumesAccessConfig(profile=os.getenv("DATABRICKS_PROFILE")),
              host=os.getenv("DATABRICKS_HOST"),
              catalog=os.getenv("DATABRICKS_CATALOG"),
              schema=os.getenv("DATABRICKS_SCHEMA"),
              volume=os.getenv("DATABRICKS_VOLUME"),
              volume_path=os.getenv("DATABRICKS_VOLUME_PATH")
          ),
          # Other examples:
          #
          # For Databricks on AWS, with Databricks personal access token authentication:
          # downloader_config=DatabricksAWSVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksAWSVolumesConnectionConfig(
          #     access_config=DatabricksAWSVolumesAccessConfig(token=os.getenv("DATABRICKS_TOKEN")),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          #
          # For Azure Databricks, with Microsoft Entra ID service principal authentication:
          # downloader_config=DatabricksAzureVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksAzureVolumesConnectionConfig(
          #     access_config=DatabricksAzureVolumesAccessConfig(
          #         azure_client_id=os.getenv("ARM_CLIENT_ID"),
          #         azure_client_secret=os.getenv("ARM_CLIENT_SECRET"),
          #         azure_tenant_id=os.getenv("ARM_TENANT_ID")   
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          #
          # For Databricks on Google Cloud, with Google Cloud Platform credentials authentication:
          # downloader_config=DatabricksGoogleVolumesDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          # source_connection_config=DatabricksGoogleVolumesConnectionConfig(
          #     access_config=DatabricksGoogleVolumesAccessConfig(
          #         google_service_account=os.getenv("GOOGLE_CREDENTIALS")
          #     ),
          #     host=os.getenv("DATABRICKS_HOST")
          # ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Delta Table
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/delta-table



Connect Delta Table to your preprocessing pipeline, and batch process all your documents using `unstructured-ingest` to
store structured outputs locally on your filesystem.

Make sure to have the Delta Table dependencies installed:

```bash Shell theme={null}
pip install "unstructured-ingest[delta-table]"
```

AWS credentials need to be available for use with the storage options.
Specify the to the DeltaTable using the `table-uri` argument, and pass a dictionary of the options to use for the storage backend
via `storage_options`.

<CodeGroup>
  ```bash Shell theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    delta-table \
      --table-uri s3://utic-dev-tech-fixtures/sample-delta-lake-data/deltatable/ \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --storage_options "AWS_REGION=us-east-2,AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" \
      --verbose \
      --strategy hi_res
  ```

  ```python Python theme={null}
  import os

  from unstructured_ingest.connector.delta_table import SimpleDeltaTableConfig
  from unstructured_ingest.interfaces import PartitionConfig, ProcessorConfig, ReadConfig
  from unstructured_ingest.runner import DeltaTableRunner

  if __name__ == "__main__":
      runner = DeltaTableRunner(
          processor_config=ProcessorConfig(
              verbose=True,
              output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"),
              num_processes=2,
          ),
          read_config=ReadConfig(),
          partition_config=PartitionConfig(
              strategy="hi_res",
          ),
          connector_config=SimpleDeltaTableConfig(
              table_uri="s3://utic-dev-tech-fixtures/sample-delta-lake-data/deltatable/",
              storage_options={
                  "AWS_REGION": "us-east-2",
                  "AWS_ACCESS_KEY_ID": os.getenv("AWS_ACCESS_KEY_ID"),
                  "AWS_SECRET_ACCESS_KEY": os.getenv("AWS_SECRET_ACCESS_KEY"),
              },
          ),
      )
      runner.run()
  ```
</CodeGroup>

For a full list of the options the Unstructured Ingest CLI accepts check `unstructured-ingest delta-table --help`.


# Discord
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/discord



Connect Discord to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A Discord [access token](https://discord.com/developers/docs/topics/oauth2).
* The target Discord [channel IDs](https://support.discord.com/hc/articles/206346498-Where-can-I-find-my-User-Server-Message-ID#h_01HRSTXPS5FMK2A5SMVSX4JW4E).

The Discord connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[discord]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DISCORD_ACCESS_TOKEN` - The Discord access token, represented by `--token` (CLI) or `token` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    discord \
      --token $DISCORD_ACCESS_TOKEN \
      --channels 475182341782896651,847950191108554762 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.discord import (
      DiscordIndexerConfig,
      DiscordDownloaderConfig,
      DiscordConnectionConfig,
      DiscordAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=DiscordIndexerConfig(
              channels=[
                  "475182341782896651",
                  "847950191108554762"
              ]
          ),
          downloader_config=DiscordDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DiscordConnectionConfig(
              access_config=DiscordAccessConfig(token=os.getenv("DISCORD_ACCESS_TOKEN"))
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Dropbox
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/dropbox



Connect Dropbox to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

1. A [Dropbox account](https://www.dropbox.com/try/teams).

2. A Dropbox app for your Dropbox account. To create a Dropbox app, do the following:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click **Create app**.<br />
   d) For **Choose an API**, select **Scoped access**.<br />
   e) For **Choose the type of access you need**, select **App folder**.<br />
   f) Enter a name for your app, and then click **Create app**.<br />
   g) On the app's **Permissions** tab, under **Files and folders**, check the boxes labelled **files.content.read** or **files.content.write** or both,
   depending on whether you want to read files, write files, or both. Then click **Submit**.<br />
   h) On the app's **Settings** tab, note the value of the **App folder name** field. This is the name of the app folder that Dropbox will create under the `Apps` top-level folder in your Dropbox
   account that the Dropbox app will use for access. If you change the value of **App folder name** field here, Dropbox will create an app folder with that name under the `Apps` top-level folder instead.<br />
   i) Under **OAuth 2**, next to **Generated access token**, click **Generate**. Copy the value of this access token. You should only click **Generate** after you have completed all of the preceding steps first.
   This is because the access token is scoped to the specific app folder and settings at the time the access token is generated. If you change the app folder name or any of the permissions later,
   you should regenerate the access token.<br />

   <Warning>
     Access tokens are valid for **only four hours** after they are created. After this four-hour period, you can no longer use the expired access token.
     Dropbox does not allow the creation of access tokens that are valid for more than four hours.

     To replace an expired access token, you must first generate a *refresh token* for the corresponding access token. To learn how to generate an access token and its corresponding refresh token,
     see [Replace an expired access token](#replace-an-expired-access-token), later in this article.

     If you do not already have the corresponding refresh token for an existing access token, or if you lose a refresh token after you generate it,
     you must generate a new access token and its corresponding refresh token.

     Instead of continualy replacing expired access tokens yourself, you can have Unstructured do it for you as needed; just supply Unstructured
     with the refresh token along with the Dropbox app's **App key** and **App secret** values.
     To learn how to supply these to Unstructured, look for mentions of "refresh token," "app key," and "app secret" in the connector settings later in this article.
   </Warning>

3. The app folder that your Dropbox app will use for access can be found in your Dropbox account under the `Apps` top-level folder. For example, if the value of the **App folder name**
   field above is `my-folder`, then the app folder that your Dropbox app will use for access can be found under `https://dropbox.com/home/Apps/my-folder`

   <img alt="The my-folder app folder under the Apps top-level folder" />

   <Warning>
     Your Dropbox app will *not* have access to upload or download files from the root of the app folder. Instead, you *must* create a subfolder inside of the app folder for your Dropbox
     app to upload or download files from. You will use the name of that subfolder when specifying your remote URL in the next step. For example, if your Dropbox app uses an app folder named `my-folder`
     for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then the subfolder that your Dropbox app will upload and download files from
     can be found under `https://dropbox.com/home/Apps/my-folder/data`

     <img alt="The data subfolder under the my-folder subfolder" />
   </Warning>

4. Note the remote URL to your subfolder inside of the app folder, which takes the format `dropbox://<subfolder-name>`. For example,
   if your Dropbox app uses an app folder named `my-folder` for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then
   the remote URL is `dropbox://data`

   <img alt="The data subfolder under the my-folder subfolder" />

## Replace an expired access token

Dropbox app access tokens are valid for **only four hours**. After this time, you can no longer use the expired access token.

To have Unstructured automatically replace expired access tokens on your behalf, do the following:

<iframe title="YouTube video player" />

1. Get the app key and app secret values for your Dropbox app. To do this:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click your Dropbox app's icon.<br />
   d) On the **Settings** tab, next to **App key**, copy the value of the app key.<br />
   e) Next to **App secret**, click **Show**, and then copy the value of the app secret.

2. Use your web browser to browse to the following URL, replacing `<app-key>` with the app key for your Dropbox app:

   ```text theme={null}
   https://www.dropbox.com/oauth2/authorize?client_id=<app-key>&response_type=code&token_access_type=offline
   ```

3. Click **Continue**.

4. Click **Allow**.

5. In the **Access code generated** tile, copy the access code that is shown.

6. Use the [curl](https://curl.se/) utility in your Terminal or Command Prompt, or use a REST API client such as
   [Postman](https://www.postman.com/product/api-client/), to make the following REST API call, replacing the following placeholders:

   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.
   * Replace `<access-code>` with the access code that you just copied.

   ```text theme={null}
   curl --location --request POST 'https://api.dropbox.com/oauth2/token' \
   --user '<app-key>:<app-secret>' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data-urlencode 'code=<access-code>' \
   --data-urlencode 'grant_type=authorization_code'
   ```

7. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token.
   * The value of `refresh_token` is the refresh token that can be used to replace this access token much faster and easier next time.
     If you lose this refresh token, you must go back to Step 2.

   For the [Unstructured Pipelines](/pipelines/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the connector settings **Refresh token** field.
   * Add the `<app-key>` value to the connector settings **App key** field.
   * Add the `<app-secret>` value to the connector settings **App secret** field.

   For the [Unstructured API](/api-reference/overview) and [Unstructured Ingest](/open-source/ingestion/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the `refresh_token` parameter.
   * Add the `<app-key>` value to the `app_key` parameter.
   * Add the `<app-secret>` value to the connector settings `app_secret` parameter.

8. If for some reason you need to manually replace the expired access token yourself instead of having Unstructured do it for you, you can use the refresh token that you just copied to get a new access token:

   * Replace `<refresh-token>` with the refresh token.
   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.

   ```text theme={null}
   curl https://api.dropbox.com/oauth2/token \
   --data refresh_token=<refresh-token> \
   --data grant_type=refresh_token \
   --data client_id=<app-key> \
   --data client_secret=<app-secret>
   ```

9. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token. In the connector, replace the old,
     expired access token value with this new, valid access token value.
   * The value of `refresh_token` is the new, valid refresh token. To replace the expired access token yourself, go back to Step 8.

The Dropbox connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[dropbox]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `DROPBOX_REMOTE_URL` - The remote URL to the target subfolder inside of the app folder for the Dropbox app, represented by `--remote-url` (CLI) or `remote_url` (Python).
* `DROPBOX_ACCESS_TOKEN` - The value of the access token for the Dropbox app that is associated with the target app folder, represented by `--token` (CLI) or `token` (Python).
  Provide this only if for some reason you do not want Unstructured to automatically refresh expired access tokens.

To have Unstructured automatically refresh expired Dropbox App access tokens on your behalf, do not provide an access token. Instead, provide the following environment variables:

* `DROPBOX_REFRESH_TOKEN` - The value of the refresh token for the corresponding access token, represented by `--refresh-token` (CLI) or `refresh_token` (Python).
* `DROPBOX_APP_KEY` - The app key for the Dropbox app, represented by `--app-key` (CLI) or `app_key` (Python).
* `DROPBOX_APP_SECRET` - The app secret for the Dropbox app, represented by `--app-secret` (CLI) or `app_secret` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    dropbox \
      --remote-url $DROPBOX_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --refresh-token $DROPBOX_REFRESH_TOKEN \
      --app-key $DROPBOX_APP_KEY \
      --app-secret $DROPBOX_APP_SECRET \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --api-key $UNSTRUCTURED_API_KEY \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.dropbox import (
      DropboxIndexerConfig,
      DropboxDownloaderConfig,
      DropboxAccessConfig,
      DropboxConnectionConfig
  )
  from unstructured_ingest.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=DropboxIndexerConfig(remote_url=os.getenv("DROPBOX_REMOTE_URL")),
          downloader_config=DropboxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=DropboxConnectionConfig(
              access_config=DropboxAccessConfig(
                  refresh_token=os.getenv("DROPBOX_REFRESH_TOKEN"),
                  app_key=os.getenv("DROPBOX_APP_KEY"),
                  app_secret=os.getenv("DROPBOX_APP_SECRET")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Elasticsearch
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/elastic-search



Connect Elasticsearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Elastic Cloud instances are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

* For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

  <iframe title="YouTube video player" />

* For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

  <iframe title="YouTube video player" />

* You will need the name of the index on the instance.

  For the destination connector, if you need to create an index, you can use for example the following `curl` command. Replace the following placeholders:

  * Replace `<host>:<port>` with the instance's host identifier and port number.

  * Replace one of the following, and remove the other:

    * For an instance's user and password combination, replace `<username>` with your Elasticsearch or Elastic Cloud instance's user name, replace `<password>` with your password, and then remove `--user "Authorization: ApiKey <your-api-key>"`.
    * For an Elastic Cloud API key, replace `<your-api-key>` with your Elastic Cloud API key, and then remove `--user "<username>:<password>"`.

  * Replace `<index-name>` with the name of the new index on the instance.

  * Replace `<index-schema>` with the schema for the index. A schema is optional; see the explanation
    following this `curl` command for more information.

  ```bash theme={null}
  curl --request PUT "<host>:<port>/<index-name>" \
  --user "<username>:<password>" \
  --user "Authorization: ApiKey <your-api-key>" \
  [--header "Content-Type: application/json" \
  --data '<index-schema>']
  ```

  To learn more, see [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

  For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
  Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
  to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  For objects in the `metadata` field that Unstructured produces and that you want to store in an Elasticsearch or Elastic Cloud index, you must create fields in your index's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    },
    "entities": {
      "items": [
        {
          "entity": "HOME FOR FUTURE",
          "type": "ORGANIZATION"
        },
        {
          "entity": "221 Queen Street, Melbourne VIC 3000",
          "type": "LOCATION"
        }
      ],
      "relationships": [
        {
          "from": "HOME FOR FUTURE",
          "relationship": "based_in",
          "to": "221 Queen Street, Melbourne VIC 3000"
        }
      ]
    }
  }
  ```

  You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
  required by Unstructured whenever you create your own index schema:

  * `element_id`
  * `record_id`, which is required by Unstructured for intelligent record updates.
  * `type`, which is not required, but highly recommended.
  * `text`
  * `embeddings` if embeddings are generated; make sure to set `dims` to the same number of dimensions as the embedding model generates.

  ```json theme={null}
  {
    "mappings": {
      "properties": {
        "element_id": {
          "type": "keyword"
        },
        "record_id": {
          "type": "text"
        },
        "text": {
          "type": "text"
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "embeddings": {
          "type": "dense_vector",
          "dims": 1536,
          "index": true,
          "similarity": "cosine"
        },
        "metadata": {
          "properties": {
            "is_extracted": {
              "type": "boolean"
            },
            "coordinates-points": {
              "type": "float"
            },
            "coordinates-system": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates-layout_width": {
              "type": "long"
            },
            "coordinates-layout_height": {
              "type": "long"
            },
            "filetype": {
              "type": "keyword"
            },
            "languages": {
              "type": "keyword"
            },
            "page_number": {
              "type": "integer"
            },
            "image_mime_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "data_source-url": {
              "type": "keyword"
            },
            "data_source-record_locator-protocol": {
              "type": "keyword"
            },
            "data_source-record_locator-remote_file_path": {
              "type": "keyword"
            },
            "entities-items": {
              "properties": {
                "entity": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "entities-relationships": {
              "properties": {
                "from": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "relationship": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "to": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
  * [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
  * [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  <iframe title="YouTube video player" />

* For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console or [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

  <Note>
    When generating an Elastic Cloud service instance's API key through the Elasticsearch Service web console, be sure after you click **Create API key** to select **Encoded** to get the API key's value.
    (The **Encoded** API key value is the Base64 encoding of the UTF-8 representation of the API key's ID and key value joined by a colon \[:].)
  </Note>

* For self-managed Elasticsearch, you will need:

  * The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).

  * If you're using basic authentication to the self-managed instance, the user's name and password.

  * If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

    <Note>
      When generating an API key for a self-managed Elasticsearch instance by calling [POST /\_security/api\_key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key), be sure to use
      the `encoded` value in the response payload. (This `encoded` value is the Base64 encoding of the UTF-8 representation of the payload's `id` and `api_key` values joined by a colon \[:]). Do not use the `api_key` value by itself.
    </Note>

  * If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

The Elasticsearch connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[elasticsearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `ELASTICSEARCH_HOST` - The hostname and port number, defined as `<hostname>:<port-number>` and represented by `--hosts` (CLI) or `hosts` (Python).
* `ELASTICSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using Elastic Cloud to connect to ElasticSearch:

* `ELASTIC_CLOUD_ID` - The instance's Cloud ID, represented by `--cloud-id` (CLI) or `cloud_id` (Python).
* `ELASTIC_CLOUD_API_KEY` - The corresponding Cloud API key value, represented by `--api-key-id` (CLI) or `api_key_id` (Python).

If you're using basic authentication to the instance:

* `ELASTICSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
* `ELASTICSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using token-based authentication to the instance instead:

* `ELASTICSEARCH_BEARER_TOKEN` - The bearer token's value, represented by `--bearer-auth` (CLI) or `bearer_auth` (Python).
* `ELASTIC_CLOUD_API_KEY_ID` - The API key's value, represented by `--api-key` (CLI) or `api_key` (Python).

If you're using certificates:

* `ELASTICSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) file, represented by `--ca-certs` (CLI) or `ca_certs` (Python).
* `ELASTICSEARCH_SSL_ASSERT_FINGERPRINT` - The certificate's fingerprint, represented by `--ssl-assert-fingerprint` or `ssl_assert_fingerprint` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    elasticsearch \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --hosts $ELASTICSEARCH_HOST \
      --username $ELASTICSEARCH_USERNAME \
      --password $ELASTICSEARCH_PASSWORD \
      --index-name $ELASTICSEARCH_INDEX_NAME \
      --ca-certs $ELASTICSEARCH_CA_CERTS \
      --ssl-assert-fingerprint $ELASTICSEARCH_SSL_ASSERT_FINGERPRINT \
      --fields 'ethnicity, director, plot' \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.elasticsearch.elasticsearch import (
      ElasticsearchIndexerConfig,
      ElasticsearchDownloaderConfig,
      ElasticsearchConnectionConfig,
      ElasticsearchAccessConfig,
      ElasticsearchUploadStagerConfig,
      ElasticsearchUploaderConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ElasticsearchIndexerConfig(index_name=os.getenv("ELASTICSEARCH_INDEX_NAME")),
          downloader_config=ElasticsearchDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=ElasticsearchConnectionConfig(
              access_config=ElasticsearchAccessConfig(
                  password=os.getenv("ELASTICSEARCH_PASSWORD"),
                  ssl_assert_fingerprint=os.getenv("ELASTICSEARCH_SSL_ASSERT_FINGERPRINT")
              ),
              hosts=[os.getenv("ELASTICSEARCH_HOST")],
              username=os.getenv("ELASTICSEARCH_USERNAME"),
              ca_certs=os.getenv("ELASTICSEARCH_CA_CERTS")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# GitHub
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/github



Connect GitHub to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* The user or organization name, and the repository name, for the target GitHub repository, for example `Unstructured-IO/docs-samples`.
* The name of a branch to access in the repository. By default, the repository's default branch is used.
* A GitHub [account](https://docs.github.com/get-started/start-your-journey/creating-an-account-on-github).
* A GitHub [personal access token](https://docs.github.com/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) for the account.
* If you are accessing a private GitHub repository, the repository must grant the appropriate access
  [permissions](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-teams-and-people-with-access-to-your-repository)
  to the GitHub account.

The GitHub connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[github]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `GITHUB_URL` - The user or organization name, and the repository name, for the target GitHub repository; for example `Unstructured-IO/docs-samples`, represented by `--url` (CLI) or `url` (Python).
* `GITHUB_BRANCH` - The name of the target branch in the repository, for example `main`, represented by `--branch` (CLI) or `branch` (Python). By default, the repository's default branch is used.
* `GITHUB_ACCESS_TOKEN` - The value of your GitHub access token, represented by `--access-token` (CLI) or `access_token` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    github \
      --access-token $GITHUB_ACCESS_TOKEN \
      --url $GITHUB_URL \
      --branch $GITHUB_BRANCH \
      --recursive \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.github import (
      GithubIndexerConfig,
      GithubDownloaderConfig,
      GithubConnectionConfig,
      GithubAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GithubIndexerConfig(
              branch=os.getenv("GITHUB_BRANCH"),
              recursive=True
          ),
          downloader_config=GithubDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GithubConnectionConfig(
              access_config=GithubAccessConfig(access_token=os.getenv("GITHUB_ACCESS_TOKEN")),
              url=os.getenv("GITHUB_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# GitLab
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/gitlab



Connect GitLab to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A [GitLab account](https://gitlab.com/users/sign_up).
* The URL for the target [GitLab repository](https://docs.gitlab.com/ee/user/project/repository/).
* A [GitLab access token](https://docs.gitlab.com/ee/security/tokens/) that allows access to the repository.
* Optionally, the name of a specific branch to access in the repository. The repository's default repository branch is used if not otherwise specified.
* Optionally, a list of file globs to limit which types of files are accepted, for example `*.html` or `*.txt`.

The GitLab connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[gitlab]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `GITLAB_REPO_URL` - The URL for the target GitLab repository, represented by `--url` (CLI) or `url` (Python).
* `GITLAB_TOKEN` - The GitLab access token that allows access to the repository, represented by `--git-access-token` (CLI) or `access_token` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    gitlab \
    --url $GITLAB_URL \
    --git-access-token $GITLAB_TOKEN \
    --git-branch v0.0.7 \
    --git-file-glob "*.html,*.txt" \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --partition-by-api \
    --api-key $UNSTRUCTURED_API_KEY \
    --partition-endpoint $UNSTRUCTURED_API_URL \
    --strategy hi_res \
    --chunking-strategy by_title \
    --embedding-provider huggingface \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  # pip install "unstructured-ingest[gitlab]"

  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.gitlab import (
      GitLabIndexerConfig,
      GitLabDownloaderConfig,
      GitLabConnectionConfig,
      GitLabAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GitLabIndexerConfig(
              path="/",
              recursive=True,
              git_branch="v0.0.7"
          ),
          downloader_config=GitLabDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GitLabConnectionConfig(
              access_config=GitLabAccessConfig(access_token=os.getenv("GITLAB_TOKEN")),
              url=os.getenv("GITLAB_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Google Cloud Storage
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/google-cloud-storage



Connect Google Cloud Storage to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

* A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
  [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:
    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```
  * For Windows:
    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

  This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

  * `Storage Object Viewer` for bucket read access.
  * `Storage Object User` for bucket write access.
  * The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

  To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
  in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

The Google Cloud Storage connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[gcs]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `GCS_SERVICE_ACCOUNT_KEY` - The Google Cloud service account key for Google Cloud Storage, represented by `--service-account-key` (CLI) or `service_account_key` (Python).
* `GCS_REMOTE_URL` - The Google Cloud Storage bucket URL, represented by `--remote-url` (CLI) or `remote_url` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    gcs \
      --service-account-key $GCS_SERVICE_ACCOUNT_KEY \
      --remote-url $GCS_REMOTE_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --strategy hi_res \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.gcs import (
      GcsIndexerConfig,
      GcsDownloaderConfig,
      GcsConnectionConfig,
      GcsAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GcsIndexerConfig(remote_url=os.getenv("GCS_REMOTE_URL")),
          downloader_config=GcsDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GcsConnectionConfig(
              access_config=GcsAccessConfig(
                  service_account_key=os.getenv("GCS_SERVICE_ACCOUNT_KEY")
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Google Drive
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/google-drive



Connect Google Drive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Google Cloud account](https://console.cloud.google.com).

* The **Google Drive API** enabled in the account.
  [Learn how](https://cloud.google.com/apis/docs/getting-started#enabling_apis).

* Within the account, a Google Cloud service account and its related `credentials.json` key file or its contents in JSON format.
  [Create a service account](https://developers.google.com/workspace/guides/create-credentials#create_a_service_account).
  [Create credentials for a service account](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the `credentials.json` key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:

    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```

  * For Windows:

    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* A Google Drive [shared folder](https://support.google.com/drive/answer/2375091) or [shared drive](https://support.google.com/a/users/answer/7212025).

* Give the service account access to the shared folder or shared drive. To do this, share the folder or drive with the service account's email address.
  [Learn how](https://support.google.com/drive/answer/7166529).
  [Learn more](https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Can-i-give-access-to-document-of-google-drive-to-service-account/m-p/530106).

* Get the shared folder's ID or shared drive's ID. This is a part of the URL for your Google Drive shared folder or shared drive, represented in the following URL as `{folder_id}`: `https://drive.google.com/drive/folders/{folder-id}`.

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "11111111111111111111"
                            ],
                            "groups": [
                                "22222222222222222222",
                                "33333333333333333333"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "44444444444444444444",
                                "55555555555555555555"
                            ],
                            "groups": [
                                "66666666666666666666",
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "77777777777777777777"
                            ],
                            "groups": [
                                "88888888888888888888"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Google Cloud user, use the user's ID along with the
[Admin SDK API](https://developers.google.com/workspace/admin/directory/reference/rest/v1/users/get) or the
[People API](https://developers.google.com/people/api/rest/v1/people/get) for Google Cloud.

To look up information about a particular Google Cloud group, use the group's ID along with the
[Admin SDK API](https://developers.google.com/workspace/admin/directory/reference/rest/v1/groups/get) or the
[Cloud Identity API](https://cloud.google.com/identity/docs/reference/rest/v1/groups/get) for Google Cloud.

The Google Drive connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[google-drive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `GOOGLE_DRIVE_FOLDER_ID` - The folder or drive ID, represented by `--drive-id` (CLI) or `drive_id` (Python).
* One of the following:

  * `GCP_SERVICE_ACCOUNT_KEY_FILEPATH` - The path to the `credentials.json` key file, represented by `--service-account-key-path` (CLI) or `service_account_key_path` (Python).
  * `GCP_SERVICE_ACCOUNT_KEY_STRING` - The contents of the `credentials.json` key file as a string, represented by `--service-account-key` (CLI) or `service_account_key` (Python).

<Note>
  To use `--extensions` with a comma-separated list (CLI) or `extensions` with an array of strings (Python) to process files with only those extensions,
  do not include the leading dot in the file extensions. For example, use `pdf` or `docx` instead of `.pdf` or `.docx`.
</Note>

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    google-drive \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --drive-id $GOOGLE_DRIVE_FOLDER_ID \
      --service-account-key-path $GCP_SERVICE_ACCOUNT_KEY_FILEPATH \ # Or
      --service-account-key $GCP_SERVICE_ACCOUNT_KEY_STRING \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.google_drive import (
      GoogleDriveConnectionConfig,
      GoogleDriveAccessConfig,
      GoogleDriveIndexerConfig,
      GoogleDriveDownloaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=GoogleDriveIndexerConfig(),
          downloader_config=GoogleDriveDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=GoogleDriveConnectionConfig(
              access_config=GoogleDriveAccessConfig(
                  service_account_key_path=os.getenv("GCP_SERVICE_ACCOUNT_KEY_FILEPATH"), # Or
                  service_account_key=os.getenv("GCP_SERVICE_ACCOUNT_KEY_STRING")
              ),
              drive_id=os.getenv("GOOGLE_DRIVE_FOLDER_ID"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>

## Output a key file's contents as a string

If you need to convert the contents of a `credentials.json` file into a string, you could use a Python script such as the following.
This script takes the local path to the key file as input and outputs the key file's contents as a string.

```python Python theme={null}
# Filename: json_file_to_string.py

import json
import sys

def json_file_to_string(file_path):
    try:
        # Read the JSON file.
        with open(file_path, 'r') as file:
            # Load the JSON data.
            data = json.load(file)
        
        # Convert the JSON data back to a string, with no whitespace.
        json_string = json.dumps(data, separators=(',', ':'))
        
        return json_string
    
    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
        return None
    except json.JSONDecodeError:
        print(f"Error: '{file_path}' is not a valid JSON file.")
        return None
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return None

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python json_file_to_string.py <path_to_json_file>")
        sys.exit(1)
    
    file_path = sys.argv[1]
    result = json_file_to_string(file_path)
    
    if result:
        print(result)
```


# Jira
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/jira



Connect Jira to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Jira Cloud account](https://www.atlassian.com/try/cloud/signup?bundle=jira-software\&edition=free) or
  [Jira Data Center installation](https://confluence.atlassian.com/adminjiraserver/installing-jira-data-center-938846870.html).
* The site URL for your [Jira Data Center installation](https://confluence.atlassian.com/jirakb/find-your-site-url-to-set-up-the-jira-data-center-and-server-mobile-app-954244798.html) or Jira Cloud account.
  For Jira Cloud, open Jira in your web browser and copy the address from the browser's address bar.
  If you're unsure, check the dashboard URL, or if viewing an issue, project or board, the site URL is typically everything that comes before **but not including** `/jira`, such as
  `https://<organization>.atlassian.net`.
* To process Jira projects, provide the IDs for the target projects. To get a project's ID, sign in to your Jira Cloud account or Jira Data Center installation, and then go to the following URL: `https://<organization>.atlassian.net/rest/api/latest/project/<project-key>`,
  replacing `<organization>` with yours, and replacing `<project-key>` with the target project's key. In the
  response, look for the URL `https://<organization>.atlassian.net/rest/api/3/project/<project-id>`, where `<project-id>` is the target project's ID.
* To process Jira boards, the IDs for the target boards. To get a board's ID, sign in to your Jira Cloud account or Jira Data Center installation, and then go to the following URL: `https://<organization>.atlassian.net/rest/agile/1.0/board?projectKeyOrId=<project-key-or-id>`,
  replacing `<organization>` with yours, and `<project-key-or-id>` with the associated project's key or ID. In the
  response, look for the URL `https://<organization>.atlassian.net/rest/agile/1.0/board/<board-id>`, where `<board-id>` is the board's ID.
* To process Jira issues, the IDs for the target issues. To get an issue's ID, sign in to your Jia Cloud account or Jira Data Center installation, open the issue, and then look at the URL in your browser's address bar. The issue ID is the string of characters after the final slash in the URL.
* A user in your [Jira Cloud account](https://support.atlassian.com/jira-cloud-administration/docs/manage-users-groups-permissions-and-roles-in-jira-cloud/) or
  [Jira Data Center installation](https://confluence.atlassian.com/adminjiraserver/create-edit-or-remove-a-user-938847025.html).
* The user must have the correct permissions in your
  [Jira Cloud account](https://support.atlassian.com/jira-cloud-administration/docs/manage-users-groups-permissions-and-roles-in-jira-cloud/) or
  [Jira Data Center installation](https://confluence.atlassian.com/jirakb/permissions-made-simple-for-jira-server-717062767.html) to
  access the target projects, boards, and issues.
* One of the following:

  * For Jira Cloud or Jira Data Center, the target user's name or email address, and password.
    [Change a Jira Cloud user's password](https://support.atlassian.com/user-management/docs/change-password-for-portal-only-customers/).
    [Change a Jira Data Center user's password](https://confluence.atlassian.com/adminjiraserver/create-edit-or-remove-a-user-938847025.html).
  * For Jira Cloud only, the target user's name or email address, and API token.
    [Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
  * For Jira Data Center only, the target user's personal access token (PAT).
    [Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).

The Jira connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[jira]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `JIRA_URL` - The site URL for your Jira Data Center installation or Jira Cloud account, represented by `--url` (CLI) or `url` (Python).
* One of the following:

  * For Jira Cloud or Jira Data Center, the target user's name or email address, and password or API token, as follows:

    * `JIRA_USERNAME` - The name or email address of the target user, represented by `--username` (CLI) or `username` (Python).
    * `JIRA_PASSWORD_OR_API_TOKEN` - The user's password (for password authentication) or API token (for API token authentication), represented by `--password` (CLI) or `password` (Python).

  * For Jira Data Center only, the target user's personal access token (PAT), as follows:

    * `JIRA_PERSONAL_ACCESS_TOKEN` - The user's personal access token (PAT), represented by `--token` (CLI) or `token` (Python).

Also, to process specific projects, boards, or issues, use:

* `--projects` with a comma-delimited list of target project IDs (CLI) or `project` with an array of target project IDs (Python).
* `--boards` with a comma-delmited list of target board IDs (CLI) or `boards` with an array of target board IDs (Python).
* `--issues` with a comma-delimited list of target issue IDs (CLI) or `issues` with an array of target issue IDs (Python).
* `--status-filter` with a comma-delimited list of statuses to filter Jira issues by (CLI) or `status_filter` with an array of statuses to filter Jira issues by (Python).
* `--download-attachments` (CLI) or `download_attachments=True` (Python) to download attachments from Jira issues.
* `--no-download-attachments` (CLI) or `download_attachments=False` (Python) to not download attachments from Jira issues. This is the default behavior.
* `--cloud` (CLI) or `cloud=True` (Python) to specify using Jira Cloud. The default is to use Jira Data Center.
* `--no-cloud` (CLI) or `cloud=False` (Python) to specify using Jira Data Center. This is the default behavior.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    jira \
      --url $JIRA_URL \
      --username $JIRA_USERNAME \
      --password $JIRA_PASSWORD_OR_API_TOKEN \ # Password or API token authentication.
      --token $JIRA_PERSONAL_ACCESS_TOKEN \ # Personal access token authentication only.
      --cloud \ # For Jira Cloud.
      --no-cloud \ # For Jira Data Center (default).
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --projects <project-id>,<project-id> \
      --boards <board-id>,<board-id> \
      --issues <issue-id>,<issue-id> \
      --status-filters <status>,<status> \
      --download-attachments \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.jira import (
      JiraIndexerConfig,
      JiraDownloaderConfig,
      JiraConnectionConfig,
      JiraAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=JiraIndexerConfig(
              projects=[
                  "<project-id>",
                  "<project-id>"
              ],
              boards=[
                  "<board-id>",
                  "<board-id>"
              ],
              issues=[
                  "<issue-id>",
                  "<issue-id>"
              ],
              status_filter=[
                  "<status>",
                  "<status>"
              ]
          ),
          downloader_config=JiraDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              download_attachments=True
          ),
          source_connection_config=JiraConnectionConfig(
              access_config=JiraAccessConfig(
                  password=os.getenv("JIRA_PASSWORD_OR_API_TOKEN"), # Password or API token authentication.
                  # token=os.getenv("JIRA_PERSONAL_ACCESS_TOKEN") # Personal access token authentication only.
              ),
              url=os.getenv("JIRA_URL"),
              username=os.getenv("JIRA_USERNAME"), # For password or API token authentication.
              cloud=True # True for Jira Cloud, False (default) for Jira Data Center.
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Kafka
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/kafka



Connect Kafka to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A Kafka cluster in [Confluent Cloud](https://www.confluent.io/confluent-cloud).
  ([Create a cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#create-ak-clusters).)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

  <iframe title="YouTube video player" />

* The [hostname and port number](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#view-a-ak-cluster) of the bootstrap Kafka cluster to connect to..

* The name of the topic to read messages from or write messages to on the cluster.
  [Create a topic](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).
  [Access available topics](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).

* For authentication, an [API key and secret](https://docs.confluent.io/cloud/current/security/authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.html#add-an-api-key).

The Kafka connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[kafka]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `KAFKA_BOOTSTRAP_SERVER` - The hostname of the bootstrap Kafka cluster to connect to, represented by `--bootstrap-server` (CLI) or `bootstrap_server` (Python).
* `KAFKA_PORT` - The port number of the cluster, represented by `--port` (CLI) or `port` (Python).
* `KAFKA_TOPIC` - The unique name of the topic to read messages from and write messages to on the cluster, represented by `--topic` (CLI) or `topic` (Python).

If you use Kafka API keys and secrets for authentication:

* `KAFKA_API_KEY` - The Kafka API key value, represented by `--kafka-api-key` (CLI) or `kafka_api_key` (Python).
* `KAFKA_SECRET` - The secret value for the Kafka API key, represented by `--secret` (CLI) or `secret` (Python).

Additional settings include:

* `--confluent` (CLI) or `confluent` (Python): True to indicate that the cluster is running Confluent Kafka.
* `--num-messages-to-consume` (CLI) or `num_messages_to_consume` (Python): The maximum number of messages to get from the topic. The default is `1` if not otherwise specified.
* `--timeout` (CLI) or `timeout` (Python): The maximum amount of time to wait for the response of a request to the topic, expressed in seconds. The default is `1.0` if not otherwise specified.
* `--group-id` (CLI) or `group_id` (Python): The ID of the consumer group, if any, that is associated with the target Kafka cluster.
  (A consumer group is a way to allow a pool of consumers to divide the consumption of data
  over topics and partitions.) The default is `default_group_id` if not otherwise specified.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    kafka \
      --bootstrap-server $KAFKA_BOOTSTRAP_SERVER \ 
      --port $KAFKA_PORT \
      --topic $KAFKA_TOPIC \
      --kafka-api-key $KAFKA_API_KEY \
      --secret $KAFKA_API_KEY \
      --confluent true \
      --batch-size 100 \
      --num-messages-to-consume 1 \
      --timeout 1.0 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.kafka.cloud import (
      CloudKafkaIndexerConfig,
      CloudKafkaDownloaderConfig,
      CloudKafkaConnectionConfig,
      CloudKafkaAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=CloudKafkaIndexerConfig(
              topic=os.getenv("KAFKA_TOPIC"),
              num_messages_to_consume=100,
              timeout=1
          ),
          downloader_config=CloudKafkaDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=CloudKafkaConnectionConfig(
              access_config=CloudKafkaAccessConfig(
                  kafka_api_key=os.getenv("KAFKA_API_KEY"),
                  secret=os.getenv("KAFKA_SECRET")
              ),
              bootstrap_server=os.getenv("KAFKA_BOOTSTRAP_SERVER"),
              port=os.getenv("KAFKA_PORT")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Local
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/local



Connect local files to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

You will need the local source connector dependencies:

```bash CLI, Python theme={null}
pip install unstructured-ingest
```

To use the local source connector, you must set `--input-path` (CLI) or `input_path` (Python) to the path in the local filesystem which contains
documents you wish to process. Optionally, you can limit processing to certain file types by setting `--file-glob` (CLI) or `file_glob` (Python), for example to `.docx`
to process only `.docx` files.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported.

The following 3-minute video shows how to use the Unstructured Ingest Python library to send multiple PDFs from a local directory in batches to be ingested by Unstructured for processing:

<iframe title="YouTube video player" />

This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    local \
      --input-path $LOCAL_FILE_INPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.local import (
      LocalIndexerConfig,
      LocalDownloaderConfig,
      LocalConnectionConfig,
      LocalUploaderConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
          downloader_config=LocalDownloaderConfig(),
          source_connection_config=LocalConnectionConfig(),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# MongoDB
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/mongodb



Connect MongoDB to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

The MongoDB requirements for a MongoDB Atlas deployment include:

<Warning>
  For MongoDB Atlas, SCRAM-SHA-1 is not supported for authentication. This means that cluster types that only
  include SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, and **Serverless**, are **not** supported.
  Unstructured only supports SCRAM-SHA-256 for MongoDB Atlas, which is cryptographically stronger than SCRAM-SHA-1.

  If you try to test or use a connector that refers to a cluster type that only includes SCRAM-SHA-1, the
  operation will fail, and you will get an error message similar to the following:
  `[digital envelope routines] unsupported`.
</Warning>

* A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).

* A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). Be sure to **not**
  select a cluster type that only includes SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, or **Serverless**.

* The cluster must be reachable from your application environment, for example by adding IP addresses to your IP access list. [Learn more](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).

* The cluster must be configured to allow IP address. [Learn how](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries).

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

* The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).

* The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).

* The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).

  <Note>
    For the destination connector, Unstructured recommends that all documents in the target collection have a field
    named `record_id` with a `String` data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the collection or, in some cases, the operation could fail altogether.
  </Note>

* The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

  ```text theme={null}
  mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
  ```

  To get the connection string in MongoDB Atlas, do the following:

  1. Log in to your MongoDB Atlas console.
  2. In the sidebar, under **Databases**, click **Clusters**.
  3. Click on the cluster you want to connect to.
  4. Click **Connect**.
  5. Click **Drivers**.
  6. Under **Add your connection string into your application code**, copy the connection string.
     You can then close the **Connect** dialog in MongoDB Atlas.

     Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

  [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

The MongoDB connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[mongodb]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

For a MongoDB Atlas deployment, the following environment variables:

* `MONGODB_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

* `MONGODB_COLLECTION` - The name of the collection in the database, represented by `--collection` (CLI) or `collection` (Python).

* `MONGODB_URI` - The connection string for the cluster, represented by `--uri` (CLI) or `uri` (Python).

For a local MongoDB server, the following environment variables:

* `MONGODB_HOST` - The host for the local MongoDB server, represented by `--host` (CLI) or `host` (Python).

* `MONGODB_PORT` - The port for the local MongoDB server, represented by `--port` (CLI) or `port` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    mongodb \
      --metadata-exclude filename,file_directory,metadata.data_source.date_processed \
      --uri $MONGODB_URI \
      --database $MONGODB_DATABASE \
      --collection $MONGODB_COLLECTION \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.mongodb import (
      MongoDBAccessConfig,
      MongoDBConnectionConfig,
      MongoDBIndexerConfig,
      MongoDBDownloaderConfig
  )
  from unstructured_ingest.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=MongoDBIndexerConfig(batch_size=100),
          downloader_config=MongoDBDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=MongoDBConnectionConfig(
              access_config=MongoDBAccessConfig(uri=os.getenv("MONGODB_URI")),
              database=os.getenv("MONGODB_DATABASE"),
              collection=os.getenv("MONGODB_COLLECTION")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Notion
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/notion



Connect Notion to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A Notion access token for an
  [internal integration](https://developers.notion.com/docs/authorization#internal-integration-auth-flow-set-up) or
  [public integration](https://developers.notion.com/docs/authorization#public-integration-auth-flow-set-up).

* The IDs of any specific target Notion [databases](https://developers.notion.com/docs/working-with-databases). To get a database's ID:

  1. Open the database as a full page in Notion.

  2. Click **Share**, and then click **Copy link**.

  3. Paste the link in your text editor. The URL uses the following format:

     ```text theme={null}
     https://www.notion.so/{workspace_name}/{database_id}?v={view_id}
     ```

  4. Find the part that corresponds to `{database_id}` in the URL you pasted. It is a 36-character string. This value is the database's ID.

     Note that if you receive a database ID by calling the Notion API, for example from the Notion search endpoint, it will contain hyphens in the UUIDv4 format. You may use either the hyphenated or un-hyphenated ID.

* The IDs of any specific target Notion [pages](https://developers.notion.com/docs/working-with-page-content). To get a page's ID:

  1. Open the page in Notion.
  2. Click **Share**, and then click **Copy link**.
  3. Paste the link in your text editor. The URL ends in a page ID. It should be a 32-character string.
  4. Format this value by inserting hyphens (`-`) in the following pattern: `8-4-4-4-12` (each number is the length of characters between the hyphens).

     Example: `1429989fe8ac4effbc8f57f56486db54` becomes `1429989f-e8ac-4eff-bc8f-57f56486db54`.

     This value is the page's ID.

The Notion connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[notion]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `NOTION_ACCESS_TOKEN` - The API key for the Notion integration, represented by `--notion-api-key` (CLI) or `notion_api_key` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional. 

  unstructured-ingest \
    notion \
      --notion-api-key $NOTION_ACCESS_TOKEN \
      --page-ids 4b1a4fff-d740-4e59-aa9f-cc5c7c2b4789,47dcbd6d-13f1-4fc1-af7d-5fc147697c00 \
      --database-ids 1382c376-5a0a-8034-a3c3-d33402979df2,f2a60ad9-c466-4b06-859d-1ab176dea8c8 \
      --recursive \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.notion.connector import (
      NotionIndexerConfig,
      NotionDownloaderConfig,
      NotionConnectionConfig,
      NotionAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=NotionIndexerConfig(
              page_ids=[
                  "4b1a4fff-d740-4e59-aa9f-cc5c7c2b4789",
                  "47dcbd6d-13f1-4fc1-af7d-5fc147697c00"
              ],
              database_ids=[
                  "1382c376-5a0a-8034-a3c3-d33402979df2", 
                  "f2a60ad9-c466-4b06-859d-1ab176dea8c8"
              ],
              recursive=True
          ),
          downloader_config=NotionDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=NotionConnectionConfig(
              access_config=NotionAccessConfig(notion_api_key=os.getenv("NOTION_ACCESS_TOKEN"))
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# OneDrive
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/one-drive



Connect OneDrive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* The OneDrive and SharePoint Online plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.

  The following video shows how to get a path:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials, and a username and password. Both authentication types require a Microsoft Entra ID app registration.

  <Note>
    The OneDrive connector does not support any other authentication methods, such as multifactor (MFA) or passwordless authentication.
  </Note>

  You will need to provide the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration that has access to the target OneDrive account, and
  the app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Files.ReadWrite.All` (if both reading and writing are needed)
  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
  * `Directory.Read.All`

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* Both authentication types also require the User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

* For username and password authentication, you will also need the password for the target UPN.

The OneDrive connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[onedrive]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `ONEDRIVE_PATH` - The path to the target folder in the OneDrive account, starting with the account's root folder (for example `my-folder/my-subfolder`), represented by `--path` (CLI) or `path` (Python).
* `ONEDRIVE_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account, represented by `--client-id` (CLI) or `client_id` (Python).
* `ONEDRIVE_CLIENT_CRED` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
* `ONEDRIVE_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).
* `ONEDRIVE_USER_PNAME` - The User Principal Name (UPN) for the OneDrive user account in Entra ID (typically the user's email address), represented by `--user-pname` (CLI) or `user_pname` (Python).
* `ONEDRIVE_USER_PASSWORD` - The password for the target UPN, represented by `--password` (CLI) or `password` (Python).
* `ONEDRIVE_AUTHORITY_URL` - The authentication token provider URL for the Entra ID app registration (the default is `https://login.microsoftonline.com` if not otherwise specified), represented by `--authority-url` (CLI) or `authority_url` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    onedrive \
      --client-id $ONEDRIVE_CLIENT_ID \
      --client-cred $ONEDRIVE_CLIENT_CRED \
      --authority-url $ONEDRIVE_AUTHORITY_URL \
      --tenant $ONEDRIVE_TENANT \
      --user-pname $ONEDRIVE_USER_PNAME \
      --password $ONEDRIVE_USER_PASSWORD \ # For username and password authentication.
      --path $ONEDRIVE_PATH \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.onedrive import (
      OnedriveIndexerConfig,
      OnedriveDownloaderConfig,
      OnedriveConnectionConfig,
      OnedriveAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OnedriveIndexerConfig(path=os.getenv("ONEDRIVE_PATH")),
          downloader_config=OnedriveDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
          ),
          source_connection_config=OnedriveConnectionConfig(
              access_config=OnedriveAccessConfig(
                  client_cred=os.getenv("ONEDRIVE_CLIENT_CRED"),
                  password=os.getenv("ONEDRIVE_USER_PASSWORD") # For username and password authentication.
              ),
              client_id=os.getenv("ONEDRIVE_CLIENT_ID"),
              tenant=os.getenv("ONEDRIVE_TENANT"),
              user_pname=os.getenv("ONEDRIVE_USER_PNAME"),
              authority_url=os.getenv("ONEDRIVE_AUTHORITY_URL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# OpenSearch
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/opensearch



Connect OpenSearch to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

## Set up OpenSearch

Supported OpenSearch installations vary by product:

* **[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):** Non-local OpenSearch only. Local OpenSearch instances are not supported.
* **[Unstructured Ingest](/open-source/ingestion/overview):** Both local and non-local OpenSearch instances are supported.

### Set up an AWS OpenSearch Service domain

To set up an [AWS OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) domain, complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.

4. Click **Create domain**.

5. In the **Name** tile, for **Domain name**, enter some unique domain name for your new OpenSearch domain.

6. In the **Domain creation method** tile, select **Easy create**. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With **Standard create**, you must enable FGAC manually.) [Standard create method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html).

7. In the **Engine options** tile, for **Version**, AWS recommends that you select the latest version.

8. In the **Network** tile, for **Network**, select a network access method.
   For faster setup, this example uses the **Public access** method.
   [VPC access method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#prerequisites-vpc-endpoints).

9. For **IP address type**, select **Dual-stack mode**.

10. In the **Fine-grained access control** (FGAC) tile, do one of the following:

    * If you want to use an existing AWS IAM user in the AWS account as the domain's master user, then for **Master user**, select **Set IAM ARN as master user**. Then enter the IAM ARN for the master user in the **IAM ARN** box.
    * If you want to create a master user and password as the domain's master user instead, then for **Master user**, select **Create master user**. Then specify some username and password for this
      new master user by filling in the **Master username**, **Master password**, and **Confirm master password** fields. Make
      sure to save the master user's password in a secure location.

11. Click **Create**.

12. After the domain is created, you must allow Unstructured to access the domain, as follows:

    a. If the new domain's settings page is not already showing, open it as follows:
    in your Amazon OpenSearch Service console, on the sidebar, expand **Managed clusters**, and then click **Domains**. Then,
    in the list of available domains, click the name of the newly created domain.<br />
    b. On the **Security configuration** tab, click **Edit**.<br />
    c. In the **Access policy** tile, for **Domain access policy**, select **Only use fine-grained access control**.<br />
    d. Click **Clear policy**. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).<br />
    e. Click **Save changes**.

### Set up an Amazon OpenSearch Serverless collection

To set up an [Amazon OpenSearch Serverless collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-create-console.html), complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Serverless**, and then click **Dashboard**.

4. Click **Create collection**.

5. In the **Collection details** tile, for **Collection name**, enter some unique name for your new OpenSearch Serverless collection.
   Optionally, for **Description**, enter some meaningful description for your new collection.

6. For **Collection type**, select **Search**.

   <Note>
     Unstructured does not support the **Vector search** collection type. If you need vector search support, you can either continue
     with these steps to use the **Search** collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead.
     However, note that the Amazon OpenSearch Serverless **Search** collection type is not as optimal as the **Vector search** collection type.
   </Note>

7. In the **Collection creation method** tile, select **Standard create**.

8. For **Encryption**, choose an AWS KMS key type.

9. For **Network access settings**, choose an **Access type**.

10. For **Resource type**, select both **Enable access to OpenSearch endpoint** and **Enable access to OpenSearch Dashboards**.

11. Click **Next**.

12. In the **Definition method** tile, select **JSON**.

13. In the **JSON editor** box, enter the following JSON, replacing the following placeholders:

    * Replace `<collection-name>` with the name of the new OpenSearch Serverless collection.
    * Replace `<account-id>` with the target AWS account ID.
    * Replace `<user-id>` with the ID of the target AWS IAM user.

    ```json theme={null}
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    ```

14. Click **Next**.

15. For **Data access policy settings**, select **Create as a new data access policy**.

16. In the **Name and description** tile, enter some unique name and an optional description for the new data access policy.

17. Click **Next**.

18. Enter any desired index details, and click **Next** again. For example:

    a. For **Index name**, enter the name of the new index in the collection.<br />
    b. For **Automatic Semantic Enrichment fields**, click **Add**, enter `embeddings` for **Automatic Semantic Enrichment field name**, click **Add**, and click **Confirm**.<br />
    c. For **Lexical search fields**, click **Add**, enter `text` for **Field name** and select **Text** for **Data type**, click **Add**, and click **Confirm**.<br />

19. Click **Submit**.

### Set up a local OpenSearch instance

The following video shows how to set up a [local OpenSearch](https://opensearch.org/downloads.html) instance.

<iframe title="YouTube video player" />

### Get the host URL

Find your host URL in the AWS console using the steps for your OpenSearch type.

#### Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.
4. In the list of available domains, click the name of your domain.
5. In the **General information** tile, copy the value of **Domain endpoint v2 (dual stack)**.

#### Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Serverless**, and then click **Dashboard**.
4. In the list of available collections, click the name of your collection.
5. On the **Overview** tab, in the **Endpoint** tile, copy the value of **OpenSearch endpoint**.

#### Local OpenSearch instance

Your local instance URL depends on how OpenSearch is installed and configured. For guidance, see [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) in the OpenSearch documentation.

### Create a search index

The name of the search index on the instance is required.

For the destination connector, if you need to create an index and you're using a master user and password as the domain's master user, you can use for example the following `curl` command. Replace the following placeholders:

* Replace `<host>` with the instance's host URL.
* Replace `<port>` with the instance's port number, which is typically `443` (for encrypted connections, and less commonly `9200` for unencrypted connections).
* Replace `<master-username>` with the master user's name, and replace `<master-password>` with the master user's password.
* Replace `<index-name>` with the name of the new search index on the instance.
* Replace `<index-schema>` with the schema for the new search index on the instance. A schema is optional; see the explanation
  following this `curl` command for more information.

```bash theme={null}
curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
```

If you're using an existing AWS IAM user as the domain's master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the preceding `curl` command. To learn how, see [create-index](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-index.html) in the AWS CLI Command Reference.

For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

For objects in the `metadata` field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index's schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
required by Unstructured whenever you create your own index schema:

* `element_id`
* `record_id`, which is required by Unstructured for intelligent record updates.
* `type`, which is not required, but highly recommended.
* `text`
* `embeddings` if embeddings are generated; make sure to set `dimension` to the same number of dimensions as the embedding model generates.

```json theme={null}
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

See also:

* [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/)
* [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
* [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
* [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
* [Unstructured document elements and metadata](/concepts/document-elements)

### Set up master user authentication

<Note>
  If you are using Enterprise Connect on a [dedicated instance](/business/dedicated-instances/overview), you do not need **master user credentials**. Skip to the **Enterprise Connect** section on this page.
</Note>

For non-local OpenSearch instances, or if you're using basic authentication to a local OpenSearch instance, you will need the master user's name and password.

For local OpenSearch instances, if you're using certificates for authentication instead of basic authentication, you will need:

* The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
* The path to the combined private key and certificate file, or
* The paths to the separate private key and certificate files.

To learn more, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

The OpenSearch connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[opensearch]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `OPENSEARCH_HOST_URL` - The OpenSearch instance's host URL, which typically takes the form of `https://<host>:<port>`, and is represented by `--hosts` (CLI) or `hosts` (Python).
* `OPENSEARCH_INDEX_NAME` - The name of the search index, represented by `--index-name` (CLI) or `index_name` (Python).

If you're using basic authentication to the instance:

* `OPENSEARCH_USERNAME` - The user's name, represented by `--username` (CLI) or `username` (Python).
* `OPENSEARCH_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).

If you're using certificates for authentication instead:

* `OPENSEARCH_CA_CERTS` - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by `--ca-certs` (CLI) or `ca_certs` (Python).
* `OPENSEARCH_CLIENT_CERT` - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by `--client-cert` (CLI) or `client_cert` (Python).
* `OPENSEARCH_CLIENT_KEY` - The path to the private key file, if `OPENSEARCH_CLIENT_CERT` refers to just the certificate file. This is represented by `--client-key` (CLI) or `client_key` (Python).

Additional related settings include:

* `--use-ssl` (CLI) or `use_ssl=True` (Python) to use SSL for the connection.
* `--verify-certs` (CLI) or `verify_certs=True` (Python) to verify SSL certificates.
* `--ssl-show-warn` (CLI) or `ssl_show_warn=True` (Python) to show a warning when verifying SSL certificates is disabled.

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    opensearch \
      --index-name $OPENSEARCH_INDEX_NAME \
      --fields "director,plot" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --hosts $OPENSEARCH_HOST_URL \
      --username $OPENSEARCH_USERNAME \
      --password $OPENSEARCH_PASSWORD \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.elasticsearch.opensearch import (
      OpensearchIndexerConfig,
      OpensearchDownloaderConfig,
      OpenSearchConnectionConfig,
      OpenSearchAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalConnectionConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OpensearchIndexerConfig(index_name=os.getenv("OPENSEARCH_INDEX_NAME")),
          downloader_config=OpensearchDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              fields=["director", "plot"]
          ),
          source_connection_config=OpenSearchConnectionConfig(
              access_config=OpenSearchAccessConfig(
                  password=os.getenv("OPENSEARCH_PASSWORD"),
                  use_ssl=True
              ),
              hosts=[os.getenv("OPENSEARCH_HOST_URL")],
              username=os.getenv("OPENSEARCH_USERNAME")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig()
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Outlook
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/outlook



Connect Outlook to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* The Outlook user's email address.
* A Microsoft Entra ID app registration in the same Azure account as the Outlook account. You will need
  this app registration's application (client) ID, client secret, and directory (tenant) ID. [Learn how](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app).
* The Entra ID app registration must have the following Graph API permission levels of the application (not delegated) type:

  * `Mail.Read`
  * `Mail.ReadBasic`
  * `User.Read.All`

  [Learn how](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions).

The Outlook connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[outlook]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `OUTLOOK_USER_EMAIL` - The Outlook user's email address, represented by `--user-email` (CLI) or `user_mail` (Python).
* `OUTLOOK_APP_CLIENT_ID` - The application (client) ID of the Microsoft Entra ID app registration that has access to the user's email account, represented by `--client-id` (CLI) or `client_id` (Python).
* `OUTLOOK_APP_CLIENT_SECRET` - The client secret for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
* `OUTLOOK_APP_TENANT` - The directory (tenant) ID of the Entra ID app registration, represented by `--tenant` (CLI) or `tenant` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Embedding is optional.

  unstructured-ingest \
    outlook \
      --user-email $OUTLOOK_USER_EMAIL \
      --outlook-folders Inbox \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --client-id $OUTLOOK_APP_CLIENT_ID \
      --client-cred $OUTLOOK_APP_CLIENT_SECRET \
      --tenant $OUTLOOK_APP_TENANT \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.outlook import (
      OutlookIndexerConfig,
      OutlookDownloaderConfig,
      OutlookConnectionConfig,
      OutlookAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Embedding is optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=OutlookIndexerConfig(
              outlook_folders=["Inbox"],
              recursive=False,
              user_email=os.getenv("OUTLOOK_USER_EMAIL")
          ),
          downloader_config=OutlookDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=OutlookConnectionConfig(
              access_config=OutlookAccessConfig(client_cred=os.getenv("OUTLOOK_APP_CLIENT_SECRET")),
              client_id=os.getenv("OUTLOOK_APP_CLIENT_ID"),
              tenant=os.getenv("OUTLOOK_APP_TENANT")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Overview
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/overview



Connect to your favorite data storage platforms for effortless batch processing of your files. We are constantly adding
new data connectors and if you don’t see your favorite platform let us know in our community Slack.

When ingesting data from a storage via a source connector, you typically store the resulting `*.json` files in a
specified output directory.

To "rehydrate" elements in JSON form into in-memory objects to further use in your application, use `elements_from_json`:

```python theme={null}
from unstructured.staging.base import elements_from_json

elements = elements_from_json(filename=path_to_json_file)
```


# PostgreSQL
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/postgresql



Connect PostgreSQL to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local PostgreSQL installations are not supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local PostgreSQL installations are supported.

The following video shows how to set up [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):

<iframe title="YouTube video player" />

The following video shows how to set up [Azure Database for PostgreSQL](https://azure.microsoft.com/products/postgresql):

<iframe title="YouTube video player" />

* A PostgreSQL instance.

  * [Create an Amazon RDS for PostgreSQL instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html).
  * [Create an Azure Database for PostgreSQL server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-deploy-on-azure-free-account).
  * [Install PostgreSQL locally](https://www.postgresql.org/docs/current/tutorial-install.html).

* The host name and port number for the instance.

  * For Amazon RDS for PostgreSQL, learn how to [get the host name and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html#postgresql-endpoint).
  * For Azure Database for PostgreSQL, learn how to [get the host](https://learn.microsoft.com/azure/postgresql/flexible-server/quickstart-create-server#get-the-connection-information). The port number is `5432`.
  * For local PostgreSQL installations, these values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.
  * For other installation types, see your PostgreSQL provider's documentation.

* Depending on your network security requirements, you might need to allow access to your instance only from specific IP addresses.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

  To learn how to allow these IP address ranges, see your PostgreSQL provider's documentation, for example with
  [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) or
  [Azure Database for PostgreSQL](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-manage-firewall-portal#create-a-firewall-rule-after-server-is-created).

  <Note>
    For Amazon RDS for PostgreSQL, Amazon recommends that you set the instance's **Public access** setting to **No** by default, as this
    approach is more secure. This means that no
    resources can connect to the instance outside of the instance's associated Virtual Private Cloud (VPC) without extra configuration.
    [Learn more](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Hiding).
    [Access an Amazon RDS instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html).

    If you must enable public access, set the instance's **Public access** setting to **Yes**, and then adjust the instance's related
    security group to allow this access.
    [Learn how](https://repost.aws/en/questions/QUxemKa9u5TV6CmLiO-r5prg/lost-public-access-to-aws-rds-postgresql-instance).

    [Troubleshoot issues with connecting to Amazon RDS instances](https://repost.aws/knowledge-center/rds-connectivity-instance-subnet-vpc).
  </Note>

* A database in the instance.

  * For Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, the default database name is `postgres` unless a custom database name was specified during the instance creation process.
  * For local PostgreSQL installations, learn how to [create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* A table in the database. Learn how to [create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

  For the destination connector, the table must have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the fields `id`, `element_id`, `record_id`, `text`, (and `embeddings`, if you are using `pgvector` and generating vector embeddings), as follows.
  `type` is an optional field, but highly recommended.

  If you are using `pgvector` and generating vector embeddings, the number of dimensions (in this example, `1536`) must match the number of dimensions for the associated embedding model that you use in any related Unstructured workflows or ingestion pipelines.

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT
    );
    ```

    ```sql PostgreSQL with pgvector theme={null}
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536)
    );
    ```
  </CodeGroup>

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  For objects in the `metadata` field that Unstructured produces and that you want to store in PostgreSQL, you must create fields in your table's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```

    ```sql PostgreSQL with pgvector  theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "embeddings" is required if you are generating vector embeddings.
    --   If you are generating embeddings, the number of dimensions in "embeddings" 
    --   must match the number of dimensions for the associated embedding model 
    --   that you use in any related Unstructured workflows or ingestion pipelines.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536),
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```
  </CodeGroup>

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  See also:

  * [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
  * [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  The following video shows how to use the `psql` utility to connect to PostgreSQL, list databases, and list and create tables:

  <iframe title="YouTube video player" />

* A user in the database, and a password for the user.

  * For Amazon RDS for PostgreSQL, learn how to [create a user](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Roles.html).
  * For Azure Database for PostgreSQL, learn how to [create a user](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-create-users).
  * For local PostgreSQL installations, learn how to [create a user](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* Database access for the user.

  * For Amazon RDS for PostgreSQL, learn how to [control user access](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Access.html).
  * For Azure Database for PostgreSQL, learn how to [control user access](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For local PostgreSQL installations, learn how to [give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).
  * For other installation types, see your PostgreSQL provider's documentation.

The PostgreSQL connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[postgres]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `PGHOST` - The host name, represented by `--host` (CLI) or `host` (Python).
* `PGPORT` - The port number, represented by `--port` (CLI) or `port` (Python).
* `PGUSER` - The username, represented by `--username` (CLI) or `username` (Python).
* `PGPASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
* `PGDATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    postgres \
      --host $PGHOST \
      --port $PGPORT \
      --database $PGDATABASE \
      --table-name elements \
      --id-column id \
      --username $PGUSER \
      --password $PGPASSWORD \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.postgres import (
      PostgresIndexerConfig,
      PostgresDownloaderConfig,
      PostgresConnectionConfig,
      PostgresAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=PostgresIndexerConfig(
              table_name="elements",
              id_column="id"
          ),
          downloader_config=PostgresDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=PostgresConnectionConfig(
              access_config=PostgresAccessConfig(password=os.getenv("PGPASSWORD")),
              host=os.getenv("PGHOST"),
              port=os.getenv("PGPORT"),
              username=os.getenv("PGUSER"),
              database=os.getenv("PGDATABASE")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# S3
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/s3



Connect S3 to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Amazon S3 requirements:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
  [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

  <img alt="Enable anonymous bucket access" />

* For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
  can be granted in one of the following ways:

  * Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
    To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
    Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

    <iframe title="YouTube video player" />

  * Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
    Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
    [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
    [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
    a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

    <Warning>
      AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
      generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
      You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
      a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

      To overwrite the expired credentials with the new set:

      * For Unstructured Pipelines, manually update the AWS Key, AWS Secret Key, and STS Token fields in Unstructured Pipelines
        for the corresponding S3 [source](/pipelines/sources/s3) or [destination](/pipelines/destinations/s3) connector.
      * For the Unstructured API, use the Unstructured API's workflow operations to call the
        [update source](/api-reference/api/source/update-source) or
        [update destination](/api-reference/api/destination/update-destination) connector operation
        for the corresponding S3 [source](/api-reference/workflow/sources/s3) or
        [destination](/api-reference/workflow/destinations/s3) connector.
      * For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
        corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
    </Warning>

* If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

The S3 connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[s3]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `AWS_S3_URL` - The path to the S3 bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.

* If the bucket does not have anonymous access enabled, provide the AWS credentials:

  * `AWS_ACCESS_KEY_ID` - The AWS access key ID for the authenticated AWS IAM user, represented by `--key` (CLI) or `key` (Python).
  * `AWS_SECRET_ACCESS_KEY` - The corresponding AWS secret access key, represented by `--secret` (CLI) or `secret` (Python).
  * `AWS_STS_TOKEN` - If required, the AWS STS session token for temporary access, represented by `--token` (CLI) or `token` (Python).

* If the bucket has anonymous access enabled for reading from the bucket, set `--anonymous` (CLI) or `anonymous=True` (Python) instead.

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Work with user-defined metadata

User-defined metadata in S3 is metadata that you can choose to set at the time that you upload a file to an S3 bucket.
User-defined metadata is specified in S3 as a set of name-value pairs. Each name-value pair begins with `x-amz-meta-` and is
followed by a unique name.

For more information about how to add or replace user-defined metadata for a file in S3, see the following:

* [Working with object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)
* [User-defined metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata)
* [Editing object metadata in the Amazon S3 console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-object-metadata.html)

Unstructured outputs any user-defined metadata that it finds for a file into the `metadata.data_source.record_locator.metadata` field of
the document elements' output for the corresponding file. For example, if Unstructured processes a file with the user-defined metadata
`x-amz-meta-mymetadata` name set to the value `myvalue`, Unstructured outputs the following into the `metadata.data_source.record_locator.metadata` field of
the document elements' output for the corresponding file:

```json theme={null}
[
    {
        "type": "...",
        "element_id": "...",
        "text": "...",
        "metadata": {
            "data_source": {
                "record_locator": {
                    "metadata": {
                        "mymetadata": "myvalue"
                    }
                }
            }
        }
    }
]
```

## Create a pipeline that uses S3 as the source

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported.

<iframe title="YouTube video player" />

This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    s3 \
      --remote-url $AWS_S3_URL \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --key $AWS_ACCESS_KEY_ID \
      --secret $AWS_SECRET_ACCESS_KEY \
      --token $AWS_STS_TOKEN \ # If using AWS STS token.
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy fast \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
    local \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig
  from unstructured_ingest.processes.connectors.fsspec.s3 import (
      S3IndexerConfig,
      S3DownloaderConfig,
      S3ConnectionConfig,
      S3AccessConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=S3IndexerConfig(remote_url=os.getenv("AWS_S3_URL")),
          downloader_config=S3DownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=S3ConnectionConfig(
              access_config=S3AccessConfig(
                  key=os.getenv("AWS_ACCESS_KEY_ID"),
                  secret=os.getenv("AWS_SECRET_ACCESS_KEY"),
                  token=os.getenv("AWS_STS_TOKEN", None)  # If using AWS STS token.
              )
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))        
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Salesforce
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/salesforce



Connect Salesforce to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Salesforce account](https://developer.salesforce.com/signup).

* Your Salesforce username. To get this username, do the following:

  1. Log in to your Salesforce account.
  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.
  3. In the **Home** tab, under **Administration**, expand **Users**, and then click **Users**.
  4. Note the **User Name** value (*not* the **Name** value) for your user.

* The names of the Salesforce categories (objects) that you want to access, specified as a comma-separated list.
  Available categories include `Account`, `Campaign`, `Case`, `EmailMessage`, and `Lead`.

* A Salesforce connected app in your Salesforce account.

  If you do not already have a Salesforce connected app, to create one, start by creating or getting the certificate (`.crt`) and private key (`.pem`) files
  that you will associate with the connected app.

  If you do not have the certificate and private key files, you can use the `openssl` utility on your local machine to create your own
  private key and self-signed certificate, if your organization allows, by running the following commands, one command at a time:

  ```bash theme={null}
  openssl genrsa -out MyPrivateKey.pem -traditional
  openssl req -new -key MyPrivateKey.pem -out MyCertificateSigningRequest.csr
  openssl x509 -req -in MyCertificateSigningRequest.csr -signkey MyPrivateKey.pem -out MyCertificate.crt -days 365
  ```

  Of course, you can change these preceding example filenames as needed. Be sure to store these generated files in a secure location.

  To create a Salesforce connected app, do the following:

  1. Log in to your Salesforce account.

  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.

  3. In the **Home** tab, under **Platform Tools**, expand **Apps**, and then click **App Manager**.

  4. Click **New Connected App**.

  5. With **Create a Connected App** selected, click **Continue**.

  6. At a minimum, fill in the following, and then click **Save**:

     * **Connected App Name**

     * **API Name** (can be the same as **Connected App Name**, but do not use spaces or punctuation)

     * **Contact Email**

     * Under **API (Enable OAuth Settings)**, check **Enable OAuth Settings**.

     * For **Callback URL**, entering `https://localhost` is okay if you won't be using this connected app for other special authentication scenarios.

     * Check **Use digital signatures**, click **Choose File**, and browse to and select your certificate (`.crt`) file.

     * For **Selected OAuth Scopes**, move the following entries from the **Available OAuth Scopes** list to the **Selected OAuth Scopes** list:

       * **Manage user data via APIs (api)**
       * **Perform requests on your behalf at any time (refresh\_token, offline\_access)**

     * Uncheck **Require Proof Key for Code Exchange (PKCE) Extension for Supported Authorization Flows**.

     * Leave **Require Secret for Web Server Flow** checked.

     * Leave **Require Secret for Refresh Token Flow** checked.

     * Check **Enable Authorization Code and Credentials Flow**.

  7. On the connected app's details page, click **Manage**, click **Edit Policies**, set the following under **OAuth Policies**, and then click **Save**:

     * Set **Permitted Users** to **All users may self-authorize**.
     * Set **IP Relaxation** to **Relax IP restrictions**.
     * Set **Refresh Token Policy** to **Refresh token is valid until revoked**.

* The OAuth consumer key (client ID) for the Salesforce connected app.

  To get the Salesforce connected app's consumer key, do the following:

  1. Log in to your Salesforce account.
  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.
  3. In the **Home** tab, under **Platform Tools**, expand **Apps**, and then click **App Manager**.
  4. In the list of apps, click the arrow next to the target connected app, and click **View**.
  5. Click **Manage Consumer Details**.
  6. Complete the on-screen security verification.
  7. Note the **Consumer Key** value.

* You must use your Salesforce account to do a one-time approval of the Salesforce connected app by using its consumer key and callback URL. To do this, while you are logged in to your
  Salesforce account, browse to the following URL, replacing `<client-id>` with the consumer key value. This URL assumes that the callback URL
  is `https://localhost`:

  ```
  https://login.salesforce.com/services/oauth2/authorize?response_type=code&client_id=<client-id>&redirect_uri=https%3A%2F%2Flocalhost
  ```

* To ensure maximum compatibility across Unstructured service offerings, you should give the contents of the private key (`.pem`) file to Unstructured as
  a string that contains the contents of the file (*not* the private key file itself).\
  To print this string suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-private-key-file>` with the path to the private key file.

  * For macOS or Linux:

    ```bash theme={null}
    cat <path-to-private-key-file>
    ```

  * For Windows:

    ```text theme={null}
    Get-Content <path-to-private-key-file>
    ```

The Salesforce connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[salesforce]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `SALESFORCE_USERNAME` - The Salesforce username that has access to the required Salesforce categories, represented by `--username` (CLI) or `username` (Python).
* `SALESFORCE_CONSUMER_KEY` - The consumer key (client ID) for the Salesforce connected app, represented by `--consumer-key` (CLI) or `consumer_key` (Python).
* `SALESFORCE_PRIVATE_KEY` - The contents of the private key (PEM) associated with the consumer key for the Salesforce connected app, represented by `--private-key` (CLI) or `private_key` (Python), or
* `SALESFORCE_PRIVATE_KEY_PATH` - The local path to the (PEM) associated with the consumer key for the Salesforce connected app, represented by `--private-key-path` (CLI) or `private_key_path` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    salesforce \
      --username $SALESFORCE_USERNAME \
      --consumer-key $SALESFORCE_CONSUMER_KEY \
      --private-key $SALESFORCE_PRIVATE_KEY \
      --categories "EmailMessage,Account,Lead,Case,Campaign" \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.salesforce import (
      SalesforceIndexerConfig,
      SalesforceDownloaderConfig,
      SalesforceConnectionConfig,
      SalesforceAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SalesforceIndexerConfig(categories=["EmailMessage", "Account", "Lead", "Case", "Campaign"]),
          downloader_config=SalesforceDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SalesforceConnectionConfig(
              access_config=SalesforceAccessConfig(
                  consumer_key=os.getenv("SALESFORCE_CONSUMER_KEY"),
                  private_key=os.getenv("SALESFORCE_PRIVATE_KEY")
              ),
              username=os.getenv("SALESFORCE_USERNAME")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SFTP
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/sftp



Connect SFTP to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* The SFTP server hostname, port, username, and password.

  SFTP servers are offered by several vendors. For example, the following video shows how to create and set up an SFTP server by using AWS Transfer Family:

  <iframe title="YouTube video player" />

* The directory path to start accessing data from, specified as `sftp://<path>/<to>/<directory>`.

See the [SFTP documentation](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[sftp]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `SFTP_HOST` - The SFTP hostname, represented by `--host` (CLI) or `host` (Python).
* `SFTP_PORT` - The SFTP port number, represented by `--port` (CLI) or `port` (Python).
* `SFTP_REMOTE_URL` - The directory path to start accessing data from, represented by `--remote-url` (CLI) or `remote_url` (Python).
* `SFTP_USERNAME` - The SFTP username, represented by `--username` (CLI) or `username` (Python).
* `SFTP_PASSWORD` - The SFTP password, represented by `--password` (CLI) or `password` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    sftp \
      --host $SFTP_HOST \
      --port $SFTP_PORT \
      --username $SFTP_USERNAME \
      --password $SFTP_PASSWORD \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --num-processes 2 \
      --recursive \
      --verbose \
      --strategy hi_res \
      --remote-url $SFTP_REMOTE_URL \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.fsspec.sftp import (
      SftpIndexerConfig,
      SftpDownloaderConfig,
      SftpConnectionConfig,
      SftpAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SftpIndexerConfig(remote_url=os.getenv("SFTP_REMOTE_URL")),
          downloader_config=SftpDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              remote_url=os.getenv("SFTP_REMOTE_URL")
          ),
          source_connection_config=SftpConnectionConfig(
              access_config=SftpAccessConfig(password=os.getenv("SFTP_PASSWORD")),
              host=os.getenv("SFTP_HOST"),
              port=os.getenv("SFTP_PORT"),
              username=os.getenv("SFTP_USERNAME"),
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              strategy="hi_res",
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()

  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SharePoint
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/sharepoint



Connect SharePoint to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<Note>
  If you are setting up the SharePoint connector for the first time, you can skip past this note.

  Previous versions of the SharePoint connector relied on SharePoint app principals for authentication. Current versions of the
  SharePoint connector no longer support these SharePoint app principals. Microsoft deprecated support for Share Point app principals on November 27, 2023.
  SharePoint app principals will no longer work for SharePoint tenants that were created on or after November 1, 2024, and they will stop working
  for all SharePoint tenants as of April 2, 2026. [Learn more](https://learn.microsoft.com/sharepoint/dev/sp-add-ins/retirement-announcement-for-azure-acs).

  Current versions of the SharePoint connector now rely on Microsoft Entra ID app registrations for authentication.

  To migrate from SharePoint app princpals to Entra ID app regisrations, replace the following settings in your existing SharePoint connector,
  as listed in the requirements following this note:

  * Replace the deprecated SharePoint app principal's application client ID value with your replacement Entra ID app registration's **Application (client) ID** value.
  * Replace the deprecated SharePoint app principal's client secret value with your replacement Entra ID app registration's **Client secret** value.
  * Add your replacement Entra ID app registration's **Directory (tenant) ID** value, token authority URL value, and the correct set of Microsoft Graph access permissions for SharePoint Online.

  If you need migration help, get assistance from our [Slack community](https://short.unstructured.io/pzw05l7), or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Note>

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  (Even if you only plan to use SharePoint Online, you still need a plan that includes OneDrive, because the SharePoint connector is built on OneDrive technology.)
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* The SharePoint Online and OneDrive plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The SharePoint Online site base URL. This URL must meet the following requirements:

  1. Starts with `https://`.
  2. Followed by any number of uppercase or lowercase letters, digits, or hyphens (`-`). This portion of the URL must contain at least
     one dot segment after the first domain label; for example, `.com`, `.sharepoint.com`, or `.co.uk`.
  3. Followed optionally by `/sites/` or `/teams/` and ends with any name that can include any number of
     uppercase or lowercase letters, digits, hyphens (`-`), or underscores (`_`); for example, `/sites/my-site` or `/teams/my_team`.

  Some examples of valid site URLs include:

  * `https://contoso.com`
  * `https://my-company.sharepoint.com`
  * `https://example.co.uk`
  * `https://contoso.sharepoint.com/sites/engineering`
  * `https://teams.example.com/teams/marketing`

  Additionally, note the following about collection-level URLs and all site URLs:

  * Site collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com/sites/<site-collection-name>`.
  * Team collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com/teams/<team-collection-name>`.
  * Root site collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com`.
  * To process all sites within a SharePoint tenant, use a site URL with a pattern similar to `https://<tenant>-admin.sharepoint.com`.

  [Learn more](https://learn.microsoft.com/microsoft-365/community/query-string-url-tricks-sharepoint-m365).

* The display name of the SharePoint Online library to use. The default is `Documents`.

* The path to the SharePoint Online library to use. By default, the root of the target library is used.
  To start from a path other than the root, enter the path that you want to use, beginning from the root. For example, to use
  the **my-folder > my-subfolder** path in the target library, you would specify `my-folder/my-subfolder`.

  The following video shows how to get the site URL and a path within the site:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials and a username and password. Both authentication types require a
  Microsoft Entra ID app registration. You will need to provide
  the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration, and the
  app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
    [Learn more](https://learn.microsoft.com/answers/questions/2116616/service-principal-access-to-sharepoint-online).

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* The token authority URL for your Microsoft Entra ID app registration. This is typically `https://login.microsoftonline.com`

* For username and password authentication, you must also provide the User Principal Name (UPN) and its password for the OneDrive account in the Microsoft Entra ID tenant. This UPN is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
                            ],
                            "groups": [
                                "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
                                "cccccccc-cccc-cccc-cccc-cccccccccccc"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "dddddddd-dddd-dddd-dddd-dddddddddddd",
                                "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee"
                            ],
                            "groups": [
                                "ffffffff-ffff-ffff-ffff-ffffffffffff"
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "11111111-1111-1111-1111-111111111111"
                            ],
                            "groups": [
                                "22222222-2222-2222-2222-222222222222"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Microsoft Entra ID user, use the user's ID along with the
[GET /users](https://learn.microsoft.com/graph/api/user-get) operation in the Microsoft Graph API.

To look up information about a particular Microsoft Entra ID group, use the group's ID along with the
[GET /groups](https://learn.microsoft.com/graph/api/group-get) operation in the Microsoft Graph API.

The SharePoint connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[sharepoint]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `SHAREPOINT_SITE_URL` - The SharePoint site URL, represented by `--site` (CLI) or `site` (Python).
* `SHAREPOINT_LIBRARY_NAME` - The display name of the SharePoint library to use, represented by `--library` (CLI) or `library` (Python). The default is `Documents`.
* `SHAREPOINT_SITE_PATH` - The path to use within the library, represented by `--path` (CLI) or `path` (Python). The default is the root of the target library. To use a different path, specify the correct path format as described previously in this article.
* `ENTRA_ID_APP_CLIENT_ID` - The **Application (client) ID** value for the Microsoft Entra ID app registration, represented by `--client-id` (CLI) or `client_id` (Python).
* `ENTRA_ID_APP_TENANT_ID` - The **Directory (tenant) ID** value for the Entra ID app registration, represented by `--client-id` (CLI) or `client_id` (Python).
* `ENTRA_ID_TOKEN_AUTHORITY_URL` - The token authority URL for the Entra ID app registration, represented by `--authority-url` (CLI) or `authority_url` (Python). The default is `https://login.microsoftonline.com`.
* `ENTRA_ID_APP_CLIENT_SECRET` - The **Client secret** value for the Entra ID app registration, represented by `--client-cred` (CLI) or `client_cred` (Python).
* `ENTRA_ID_USER_PRINCIPAL_NAME` - For username and password authentication, the User Principal Name (UPN) for the target OneDrive account in the Microsoft Entra ID tenant.
* `ENTRA_ID_USER_PASSWORD` - For username and password authentication, the password for the target UPN.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured-ingest \
    sharepoint \
      --site $SHAREPOINT_SITE_URL \
      --library $SHAREPOINT_LIBRARY_NAME \
      --path $SHAREPOINT_SITE_PATH \
      --recursive \
      --client-id $ENTRA_ID_APP_CLIENT_ID \
      --tenant $ENTRA_ID_APP_TENANT_ID \
      --authority-url $ENTRA_ID_TOKEN_AUTHORITY_URL \
      --client-cred $ENTRA_ID_APP_CLIENT_SECRET \
      --user-pname $ENTRA_ID_USER_PRINCIPAL_NAME \
      --password $ENTRA_ID_USER_PASSWORD \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy hi_res \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}"
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sharepoint import (
      SharepointIndexerConfig,
      SharepointDownloaderConfig,
      SharepointConnectionConfig,
      SharepointAccessConfig
  )

  from unstructured_ingest.processes.connectors.local import (
      LocalUploaderConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SharepointIndexerConfig(
              path=os.getenv("SHAREPOINT_SITE_PATH"),
              recursive=True  # True to recursively download files in their respective folders.
          ),
          downloader_config=SharepointDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SharepointConnectionConfig(
              access_config=SharepointAccessConfig(
                  client_cred=os.getenv("ENTRA_ID_APP_CLIENT_SECRET"),
                  password=os.getenv("ENTRA_ID_USER_PASSWORD"), # For username and password authentication.
              ),
              site=os.getenv("SHAREPOINT_SITE_URL"),
              library=os.getenv("SHAREPOINT_LIBRARY_NAME"),
              client_id=os.getenv("ENTRA_ID_APP_CLIENT_ID"),
              tenant=os.getenv("ENTRA_ID_APP_TENANT_ID"),
              authority_url=os.getenv("ENTRA_ID_TOKEN_AUTHORITY_URL"),
              user_pname=os.getenv("ENTRA_ID_USER_PRINCIPAL_NAME") # For username and password authentication.            
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "reprocess": True,
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SingleStore
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/singlestore



Connect SingleStore to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A SingleStore deployment, database, and table. [Learn how](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/).
* The hostname for the SingleStore deployment.
* The port for the host.
* The username for the deployment.
* The password for the user.
* The name of the database in the deployment.
* The name of the table in the database.

To get the values for the hostname, port, username, and password:

1. In your SingleStore account's dashboard sidebar, click **Deployments**.
2. From the drop-down list at the top of the **Deployments** page, select your deployment.
3. On the **Overview** tab, in the **Compute** area, in the **Connect** drop-down list for your deployment, select
   **Your App**.
4. If a **Create User** dialog box appears, note the **User** name and **Password** values.
5. In the **Connect to Workspace** pane's **Your App** tab, note the string in the following format:

```
<user-name>:<password>@<host>:<port>
```

* `<user-name>` is the username.
* `<password>` is the user's password.
* `<host>` is the workspace's hostname.
* `<post>` is the host's port.

To get the values for the database and table names:

1. In your SingleStore dashboard's sidebar, click **Deployments**.
2. From the drop-down list at the top of the **Deployments** page, select your deployment.
3. On the **Databases** tab, note **Name** of your database.
4. Click the database and, on the **Tables** tab, note the name of your table.

The SingleStore connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[singlestore]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `SINGLESTORE_HOST` - The hostname for the SingleStore deployment, represented by `--host` (CLI) or `host` (Python).
* `SINGLESTORE_PORT` - The port for the host, represented by `--port` (CLI) or `port` (Python).
* `SINGLESTORE_USER` - The username for the deployment, represented by `--user` (CLI) or `user` (Python).
* `SINGLESTORE_PASSWORD` - The password for the user, represented by `--password` (CLI) or `password` (Python).
* `SINGLESTORE_DB` - The name of the database in the deployment, represented by `--database` (CLI) or `database` (Python).
* `SINGLESTORE_TABLE` - The name of the table in the database, represented by `--table-name` (CLI) or `table_name` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    singlestore \
      --host $SINGLESTORE_HOST \
      --port $SINGLESTORE_PORT \
      --user $SINGLESTORE_USER \
      --password $SINGLESTORE_PASSWORD \
      --database $SINGLESTORE_DB \
      --table-name $SINGLESTORE_TABLE \
      --id-column id \
      --batch-size 100 \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --strategy fast \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.singlestore import (
      SingleStoreIndexerConfig,
      SingleStoreDownloaderConfig,
      SingleStoreConnectionConfig,
      SingleStoreAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalUploaderConfig,
      LocalConnectionConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SingleStoreIndexerConfig(
              table_name=os.getenv("SINGLESTORE_TABLE"),
              id_column="id",
              batch_size=100
          ),
          downloader_config=SingleStoreDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")
          ),
          source_connection_config=SingleStoreConnectionConfig(
              access_config=SingleStoreAccessConfig(
                  password=os.getenv("SINGLESTORE_PASSWORD")
              ),
              host=os.getenv("SINGLESTORE_HOST"),
              port=os.getenv("SINGLESTORE_PORT"),
              user=os.getenv("SINGLESTORE_USER"),
              database=os.getenv("SINGLESTORE_DB")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Slack
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/slack



Connect Slack to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Slack app. Create a Slack app by following [Step 1: Creating an app](https://api.slack.com/quickstart#creating).

* The app must have the following OAuth scopes:

  * To read messages from public channels, add the scopes `channels:join` and `channels:history`.
  * To read messages from private channels, add the scope `groups:history`.

  Add these scopes to the app by following [Step 2: Requesting scopes](https://api.slack.com/quickstart#scopes).

* The app must be installed and authorized for the target Slack workspace. Install and authorize the app by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).

* The app's access token. Get this token by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).

* Add the app to the target channels in the Slack workspace. To do this from the channel, open the channel's details page, click the **Integrations** tab, click **Add apps**, and follow the on-screen directions to install the app.

* The channel ID for each target channel. To get this ID, open the channel's details page, and look for the **Channel ID** field on the **About** tab.

* The starting and ending date and time range for the channels to be processed. Supported formats include:

  * `YYYY-MM-DD`
  * `YYYY-MM-DDTHH:MM:SS`
  * `YYYY-MM-DDTHH:MM:SSZ`
  * `YYYY-MM-DD+HH:MM:SS`
  * `YYYY-MM-DD-HH:MM:SS`

<Info>
  Slack has established API rate limits that could restrict the number and frequency of messages that can be read. To
  explore possible workarounds for these cases, see your Slack representative.
</Info>

The Slack connector dependencies:

```bash theme={null}
pip install "unstructured-ingest[slack]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `SLACK_BOT_USER_OAUTH_TOKEN` - The OAuth token for the Slack app, represented by `--token` (CLI) or `token` (Python).

To specify the starting and ending date and time range for the channels to be processed:

* For the CLI, use one of the following supported formats:

  * `YYYY-MM-DD`
  * `YYYY-MM-DDTHH:MM:SS`
  * `YYYY-MM-DDTHH:MM:SSZ`
  * `YYYY-MM-DD+HH:MM:SS`
  * `YYYY-MM-DD-HH:MM:SS`

* For Python, use the `datetime.datetime` function.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    slack \
      --token $SLACK_BOT_USER_OAUTH_TOKEN \
      --channels C03FVNHR70A,C03FVNRG43D \
      --start-date 2024-10-22 \
      --end-date 2024-10-23 \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os
  from datetime import datetime

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.slack import (
      SlackIndexerConfig,
      SlackDownloaderConfig,
      SlackConnectionConfig,
      SlackAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SlackIndexerConfig(
              channels=["C03FVNHR70A", "C03FVNRG43D"],
              start_date=datetime(year=2024, month=10, day=22),
              end_date=datetime(year=2024, month=10, day=23)
          ),
          downloader_config=SlackDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SlackConnectionConfig(
              access_config=SlackAccessConfig(token=os.getenv("SLACK_BOT_USER_OAUTH_TOKEN"))
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Snowflake
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/snowflake



Connect Snowflake to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.

  <iframe title="YouTube video player" />

  To get the identifier for the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
  3. On the **Account** tab, note the value of the **Account Identifier** field.

  Alternatively, the following Snowflake query returns the current account's identifier:

  ```text theme={null}
  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  ```

* A Snowflake user, which can be a service user (recommended) or a human user.

  To create a service user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Projects > Worksheets**.

  3. Click the **+** button to create a SQL worksheet.

  4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:

     * Replace `<service-user-name>` with some name for the service user.
     * Replace `<default-role-name>` with the name of any default role for the service user to use.

     ```sql theme={null}
     CREATE USER <service-user-name>
     DEFAULT_ROLE = "<default-role-name>"
     TYPE = SERVICE
     ```

  5. Click the arrow icon to run the worksheet, which creates the service user.

  6. To get their login name, on the navigation menu, click **Admin > Users & Roles**.

  7. On the **Users** tab, in the list of available users, click the name of the target user.

  8. In the **About** tile, note the **Login Name** for the user.

  To create a human user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & roles**.
  3. Click the **Users** tab.
  4. Click **+ User**.
  5. Follow the on-screen guidance to specify the user's settings.
  6. Click **Create User**.
  7. To get their login name, on the navigation menu, click **Admin > Users & Roles**.
  8. On the **Users** tab, in the list of available users, click the name of the target user.
  9. In the **About** tile, note the **Login Name** for the user.

* A programmatic access token (PAT) for the Snowflake user.

  <iframe title="YouTube video player" />

  To create a programmatic access token (PAT) for a user:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.

  3. On the **Users** tab, in the list of available users, click the name of the target user.

  4. In the **Programmatic access tokens** tile, click the **Generate new token** button.

  5. Follow the on-screen guidance to specify the PAT's settings.

     <Warning>
       You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later.
       Once this PAT expires, the connector will stop working.
       To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and
       update your connector's settings with your new PAT's value.

       Unstructured does not notify you when a PAT is about to expire or has already expired.
       You are responsible for tracking your PATs' expiration dates and taking corrective action before they expire.
     </Warning>

  6. Click **Generate**.

  7. Copy the generated PAT's value to a secure location, as you will not be able to access it again. If you lose this PAT's value,
     you will need to repeat this procedure to generate a new, replacement one.

  The PAT will not work unless the Snowflake account also has a valid
  [network rule](https://docs.snowflake.com/en/user-guide/network-rules) along with a valid
  [network policy](https://docs.snowflake.com/en/user-guide/network-policies) attached to that rule.
  The network rule must also be activated on the Snowflake account to begin taking effect.

  To create a valid network rule:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Rules**.

  3. Click **+ Network Rule**.

  4. Enter some name for the network rule.

  5. For **Type**, select **IPv4**.

  6. For **Mode**, select **Ingress**.

  7. For **Identifiers**, next to the magnifying glass icon, enter `0.0.0.0/0`, and then press **Enter**.

     <Note>
       The `0.0.0.0/0` value allows all IP addresses to access the Snowflake account.
       You can specify a more specific IP address range if you prefer. However, this more specific IP address range
       will apply to all users, including the user for which you created the PAT.
     </Note>

  8. Click **Create Network Rule**.

  To create a valid network policy, attaching the preceding network rule to this policy at the same time:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click **+ Network Policy**.
  4. Enter some name for the network policy.
  5. Make sure **Allowed** is selected.
  6. In the **Select rule** drop-down list, select the precedingnetwork rule to attach to this network policy.
  7. Click **Create Network Policy**.

  To activate the network rule in the account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click the name of the precedingnetwork policy to activate.
  4. In the policy's side panel, click the ellipsis (three dots) icon, and then click **Activate On Account**.
  5. Click **Activate policy**.

* (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake [user's login name (not username) and the user's password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
  This user must be a human user. Passwords are not supported for service users.

* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  <iframe title="YouTube video player" />

  * To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
  * To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
  * To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
  * To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
  * To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
  3. Click the **Roles** tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  ```text theme={null}
  SHOW ROLES;
  ```

  [Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).

* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

  <iframe title="YouTube video player" />

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:

  ```text theme={null}
  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  ```

* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

  <iframe title="YouTube video player" />

  To view a list of available databases in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  ```text theme={null}
  SHOW DATABASES;
  ```

* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  ```text theme={null}
  SHOW SCHEMAS;
  ```

  The following Snowflake query returns a list of available schemas for the database named `<database_name>` in the current account:

  ```text theme={null}
  SHOW SCHEMAS IN DATABASE <database_name>;
  ```

* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand **Tables**.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named `<schema_name>` in the datbase named
  `<database_name>` in the current account:

  ```text theme={null}
  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  ```

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the columns `ID`, `ELEMENT_ID`, and `RECORD_ID`. The columns `TEXT` and `TYPE` are optional, but highly recommended.
  If you are generating embeddings, then you must also include the column `EMBEDDINGS`.

  In the following `CREATE TABLE` statement, replace the following placeholders with the appropriate values:

  * `<database_name>`: The name of the target database in the Snowflake account.
  * `<schema_name>`: The name of the target schema in the database.
  * `<number-of-dimensions>`: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search,
    this value must match the number of dimensions that you plan to have Snowflake generate or search against.

  ```sql SQL theme={null}
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      ELEMENT_ID VARCHAR,
      RECORD_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>)
  );
  ```

  For objects in the `metadata` field that Unstructured produces and that you want to store in a Snowflake table, you must create columns in your table's schema that
  follow Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  ```sql theme={null}
  -- The ID, RECORD_ID, and ELEMENT_ID columns are required.
  -- TEXT and TYPE are not required but highly recommended.
  -- EMBEDDINGS is required if embeddings are being generated.
  -- All other "metadata" columns are optional.
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      RECORD_ID VARCHAR,
      ELEMENT_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
      IS_EXTRACTED VARCHAR,
      POINTS VARCHAR,
      SYSTEM VARCHAR,
      LAYOUT_WIDTH INTEGER,
      LAYOUT_HEIGHT INTEGER,
      FILETYPE VARCHAR,
      LANGUAGES ARRAY,
      PAGE_NUMBER VARCHAR,
      IMAGE_MIME_TYPE VARCHAR,
      FILENAME VARCHAR,
      URL VARCHAR,
      RECORD_LOCATOR VARCHAR
  );
  ```

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).

The Snowflake connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[snowflake]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `SNOWFLAKE_ACCOUNT` - The ID of the target Snowflake account, represented by `--account` (CLI) or `account` (Python).

* `SNOWFLAKE_USER`  - The name of the target Snowflake user, represented by `--user` (CLI) or `user` (Python).

* `SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN` - The user's programmatic access token (PAT), represented by `--password` (CLI) or `password` (Python).

  <Note>
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  </Note>

* `SNOWFLAKE_ROLE` - The target role for the user, represented by `--role` (CLI) or `role` (Python).

* `SNOWFLAKE_HOST`  - The hostname for the target Snowflake warehouse, represented by `--host` (CLI) or `host` (Python).

* `SNOWFLAKE_PORT`  - The warehouse's port number, represented by `--port` (CLI) or `port` (Python).  The default is `443` if not otherwise specified.

* `SNOWFLAKE_DATABASE`  - The name of the target Snowflake database, represented by `--database` (CLI) or `database` (Python).

* `SNOWFLAKE_SCHEMA` - The name of the target schema in the database, represented by `--schema` (CLI) or `schema` (Python).

* `SNOWFLAKE_TABLE` - The name of the target table in the schema, represented by `--table-name` (CLI) or `table_name` (Python). For the destination connector, the default is `elements` if not otherwise specified.

* `SNOWFLAKE_RECORD_ID_KEY` - The name of the column in the table that uniquely identifies each record, represented by:

  * For the source connector, `--id-column` (CLI) or `id_column` (Python).
  * For the destination connector, `--record-id-key` (CLI) or `record_id_key` (Python). For the destination connector, the default is `record_id` if not otherwise specified.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  unstructured \
    snowflake \
      --account $SNOWFLAKE_ACCOUNT \
      --user $SNOWFLAKE_USER \
      --password $SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN \
      --role $SNOWFLAKE_ROLE \
      --host $SNOWFLAKE_HOST \
      --port $SNOWFLAKE_PORT \
      --database $SNOWFLAKE_DATABASE \
      --schema $SNOWFLAKE_SCHEMA \
      --batch-size 50 \
      --table-name $SNOWFLAKE_TABLE \
      --id-column $SNOWFLAKE_RECORD_ID_KEY \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR\
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --output-dir $LOCAL_FILE_OUTPUT_DIR
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.snowflake import (
      SnowflakeIndexerConfig,
      SnowflakeConnectionConfig,
      SnowflakeAccessConfig,
      SnowflakeDownloaderConfig
  )
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SnowflakeIndexerConfig(
              table_name=os.getenv("SNOWFLAKE_TABLE"),
              id_column=os.getenv("SNOWFLAKE_RECORD_ID_KEY"),
              batch_size=100
          ),
          downloader_config=SnowflakeDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SnowflakeConnectionConfig(
              access_config=SnowflakeAccessConfig(
                  password=os.getenv("SNOWFLAKE_PROGRAMMATIC_ACCESS_TOKEN")
              ),
              account=os.getenv("SNOWFLAKE_ACCOUNT"),
              user=os.getenv("SNOWFLAKE_USER"),
              host=os.getenv("SNOWFLAKE_HOST"),
              port=os.getenv("SNOWFLAKE_PORT"),
              database=os.getenv("SNOWFLAKE_DATABASE"),
              schema=os.getenv("SNOWFLAKE_SCHEMA"),
              role=os.getenv("SNOWFLAKE_ROLE")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# SQLite
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/sqlite



Connect SQLite to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A SQLite instance. [Download and install SQLite](https://www.sqlitetutorial.net/download-install-sqlite/).
* A SQLite database. [Create a database](https://www.sqlite.org/quickstart.html).
* The path to the database's `.db` file.
* A table in the database. [Create a table](https://www.sqlite.org/cli.html).

  The table's schema must match the schema of the documents that Unstructured produces.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  You can adapt the following table schema example for your own needs:

  ```sql SQLite theme={null}
  CREATE TABLE elements (
      id TEXT PRIMARY KEY,
      record_id TEXT,
      element_id TEXT,
      text TEXT,
      embeddings TEXT,
      parent_id TEXT,
      page_number INTEGER,
      is_continuation INTEGER,
      orig_elements TEXT,
      partitioner_type TEXT
  );
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) for SQLite
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `SQLITE_DB_PATH` - The path to the database's `.db` file, represented by `--database` (CLI) or `database` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    sqlite \
      --database-path $SQLITE_DB_PATH \
      --table-name elements \
      --id-column id \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.sqlite import (
      SQLiteIndexerConfig,
      SQLiteDownloaderConfig,
      SQLiteConnectionConfig,
      SQLiteAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig

  from unstructured_ingest.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=SQLiteIndexerConfig(
              table_name="elements",
              id_column="id"
          ),
          downloader_config=SQLiteDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=SQLiteConnectionConfig(
              access_config=SQLiteAccessConfig(),
              database_path=os.getenv("SQLITE_DB_PATH")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Teradata
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/teradata-sql



Connect Teradata to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

* A Teradata Vantage system that can be accessed by its host name or IP address.

  For example, a Teradata Vantage system in Teradata ClearScape Analytics Experience includes:

  * A Teradata ClearScape Analytics Experience account.
  * An environment in the account.
  * A Teradata Vantage database in the environment.
  * The name and password for a Teradata user who has the appropriate access to the database.

  [Learn how to create these in Teradata ClearScape Analytics Experience](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/).

* The system's corresponding host name or IP address.

  For example, you can get these values from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics Experience account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Host** value.<br />

* The name of the target database in the system. To get a list of available databases in the system, you can run a Teradata SQL query such as the following:

  ```sql theme={null}
  SELECT DatabaseName 
  FROM DBC.DatabasesV 
  ORDER BY DatabaseName;
  ```

* The name of the target table in the database. To get a list of available tables in a database, you can run a Teradata SQL query such as the following, replacing `<database-name>` with the name of the target database:

  ```sql theme={null}
  SELECT TableName
  FROM DBC.TablesV
  WHERE DatabaseName = '<database-name>' AND TableKind = 'T'
  ORDER BY TableName;
  ```

  When Unstructured writes rows to a table, the table's columns must have a schema that is compatible with Unstructured.
  Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances.
  This is because these schemas will vary based on
  your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
  any custom post-processing code that you run; and other factors.

  In any case, note the following about table schemas:

  * The following columns are always required by Unstructured: `record_id` and `element_id`.
  * The following columns are optional for Unstructured, but highly recommended: `text` and `type`.
  * The rest of the columns are optional and typically will be output by Unstructured as part of the `metadata` field.
  * If Unstructured is generating vector embeddings, the `embeddings` column is also required.

    <Warning>
      The destination connector outputs Unstructured-generated [embeddings](/concepts/embedding) that are not directly compatible
      with Teradata Enterprise Vector Store. To use embeddings with Teradata Enterprise Vector Store, Unstructured
      recommends that you choose from among the following options:

      * Define a column in your target table named `embeddings` that is of type `VARCHAR(64000)`, to store the
        Unstructured-generated embeddings. After Unstructured adds its embeddings to your `embeddings` column,
        choose from among Teradata's options to convert the `embeddings` column's `VARCHAR` values to the
        Teradata [VECTOR Data Type](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/VECTOR-Data-Type)
        yourself.
      * Omit any columns named `embedding`, `message`, or `num_tokens` from your target table. Then choose from
        among Teradata's options (such as [AI\_TextEmbeddings](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/In-Database-Analytic-Functions-for-Vector-Store))
        to have Teradata generate the embeddings for you, instead of having Unstructured generate them.
    </Warning>

  Here is an example table schema that is compatible with Unstructured. It includes all of the required and recommended columns, as
  well as a few additional columns that are typically output by Unstructured as part of the `metadata` field. Be sure to replace
  `<database-name>` with the name of the target database and `<table-name>` with the name of the target table (by Unstructured convention,
  the table name is typically `elements`, but this is not a requirement).

  ```sql theme={null}
  CREATE SET TABLE "<database-name>"."<table-name>" (
    "id" VARCHAR(64) NOT NULL,
    PRIMARY KEY ("id"),
    "record_id" VARCHAR(64),
    "element_id" VARCHAR(64),
    "text" VARCHAR(32000) CHARACTER SET UNICODE,
    "type" VARCHAR(50),
    "embeddings" VARCHAR(64000), -- Add this column only if Unstructured is generating vector embeddings.
    "last_modified" VARCHAR(50),
    "languages" VARCHAR(200),
    "file_directory" VARCHAR(500),
    "filename" VARCHAR(255),
    "filetype" VARCHAR(50),
    "record_locator" VARCHAR(1000),
    "date_created" VARCHAR(50),
    "date_modified" VARCHAR(50),
    "date_processed" VARCHAR(50),
    "permissions_data" VARCHAR(1000),
    "filesize_bytes" INTEGER,
    "parent_id" VARCHAR(64)
  )
  ```

* For the source connector, the name of the primary key column in the table (for example, a column named `id`, typically defined as `"id" VARCHAR(64) NOT NULL, PRIMARY KEY ("id")`).

* For the source connector, the names of any specific columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* For the destination connector, the name of the column in the table that uniquely identifies each record for Unstructured to perform any necessary record updates. By default convention, Unstructured expects this field to be named `record_id`.

* The name of the Teradata user who has the appropriate access to the target database.

  For example, you can get this from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Username** value.<br />

  The Teradata SQL command to get a list of available users is as follows:

  ```sql theme={null}
  SELECT UserName 
  FROM DBC.UsersV 
  ORDER BY UserName;
  ```

* The password for the user, which was set up when the user was created.

  If the user has forgotten their password, the Teradata SQL command to change a user's password is as follows, replacing `<user-name>` with the name of the user and `<new-password>` with the new password:

  ```sql theme={null}
  MODIFY USER <user-name> SET PASSWORD = '<new-password>';
  ```

  To change a user's password, you must be an administrator (such as the `DBC` user or another user with `DROP USER` privileges).

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

The Teradata connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[teradata]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

These environment variables:

* `TERADATA_HOST` - The host name, represented by `--host` (CLI) or `host` (Python).
* `TERADATA_PORT` - The port number, represented by `--dbs-port` (CLI) or `dbs_port` (Python). This is optional, and the default is `1025` if not otherwise specified.
* `TERADATA_USERNAME` - The name of the user who has access to the database, represented by `--user` (CLI) or `user` (Python).
* `TERADATA_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
* `TERADATA_DATABASE` - The name of the database, represented by `--database` (CLI) or `database` (Python). If not otherwise specified, the default database name is used. To get the name of the default database, you can run the Teradata SQL command `SELECT DATABASE;`.
* `TERADATA_TABLE` - The name of the table, represented by `--table-name` (CLI) or `table_name` (Python).
* `TERADATA_ID_COLUMN` - For the source connector, the name of the column that uniquely identifies each record in the table, represented by `--id-column` (CLI) or `id_column` (Python).
* `TERADATA_RECORD_ID_KEY` - For the destination connector, the name of the column in the table that uniquely identifies each record, represented by `--record-id-key` (CLI) or `record_id_key` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    teradata \
      --host $TERADATA_HOST \
      # --dbs-port $TERADATA_PORT \
      --database $TERADATA_DATABASE \
      --user $TERADATA_USERNAME \
      --password $TERADATA_PASSWORD \
      --table-name $TERADATA_TABLE \
      --id-column $TERADATA_ID_COLUMN \
      --batch-size 50 \
      --fields "<column-name>,<column-name>" \
      --download-dir $LOCAL_FILE_DOWNLOAD_DIR \
      --chunking-strategy by_title \
      --embedding-provider huggingface \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL
  ```

  ```python Python SDK theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.sql.teradata import (
      TeradataIndexerConfig,
      TeradataDownloaderConfig,
      TeradataConnectionConfig,
      TeradataAccessConfig
  )
  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import (
      LocalConnectionConfig,
      LocalUploaderConfig
  )

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=TeradataIndexerConfig(
              table_name=os.getenv("TERADATA_TABLE"),
              id_column=os.getenv("TERADATA_ID_COLUMN"),
              batch_size=50
          ),
          downloader_config=TeradataDownloaderConfig(
              download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR"),
              fields=[
                  "<column-name>",
                  "<column-name>"
              ]
          ),
          source_connection_config=TeradataConnectionConfig(
              access_config=TeradataAccessConfig(
                  password=os.getenv("TERADATA_PASSWORD")
              ),
              host=os.getenv("TERADATA_HOST"),
              # dbs_port=os.getenv("TERADATA_PORT"),
              user=os.getenv("TERADATA_USERNAME"),
              database=os.getenv("TERADATA_DATABASE")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          destination_connection_config=LocalConnectionConfig(),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Zendesk
Source: https://docs.unstructured.io/open-source/ingestion/source-connectors/zendesk



Connect Zendesk to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Zendesk account](https://www.zendesk.com/register/).
* Your [Zendesk subdomain](https://support.zendesk.com/hc/en-us/articles/4409381383578-Where-can-I-find-my-Zendesk-subdomain), for example, the `<organization>` part of `<organization>.zendesk.com`.
* The login email address for your Zendesk account.
* An [API token](https://support.zendesk.com/hc/en-us/articles/4408889192858-Managing-access-to-the-Zendesk-API#topic_tcb_fk1_2yb) (not an OAuth token) for your login email address.

The Zendesk connector dependencies:

```bash CLI, Python theme={null}
pip install "unstructured-ingest[zendesk]"
```

You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).

The following environment variables:

* `ZENDESK_SUBDOMAIN` - The subdomain for your Zendesk account, represented by `--subdomain` (CLI) or `subdomain` (Python). This subdomain is the `<organization>` part of `<organization>.zendesk.com`.
* `ZENDESK_EMAIL` - The login email address for your Zendesk account, represented by `--email` (CLI) or `email` (Python).
* `ZENDESK_API_TOKEN` - The API token (not OAuth token) for your login email address, represented by `--api-token` (CLI) or `api_token` (Python).

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector.

This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.

<CodeGroup>
  ```bash CLI theme={null}
  #!/usr/bin/env bash

  # Chunking and embedding are optional.

  unstructured-ingest \
    zendesk \
      --subdomain $ZENDESK_SUBDOMAIN \
      --email $ZENDESK_EMAIL \
      --api-token $ZENDESK_API_TOKEN \
      --batch-size 2 \
      --item-type tickets \
      --output-dir $LOCAL_FILE_OUTPUT_DIR \
      --partition-by-api \
      --api-key $UNSTRUCTURED_API_KEY \
      --partition-endpoint $UNSTRUCTURED_API_URL \
      --chunking-strategy by_title \
      --embedding-provider huggingface
  ```

  ```python Python Ingest theme={null}
  import os

  from unstructured_ingest.pipeline.pipeline import Pipeline
  from unstructured_ingest.interfaces import ProcessorConfig

  from unstructured_ingest.processes.connectors.zendesk import (
      ZendeskIndexerConfig,
      ZendeskDownloaderConfig,
      ZendeskConnectionConfig,
      ZendeskAccessConfig
  )

  from unstructured_ingest.processes.partitioner import PartitionerConfig
  from unstructured_ingest.processes.chunker import ChunkerConfig
  from unstructured_ingest.processes.embedder import EmbedderConfig
  from unstructured_ingest.processes.connectors.local import LocalUploaderConfig

  # Chunking and embedding are optional.

  if __name__ == "__main__":
      Pipeline.from_configs(
          context=ProcessorConfig(),
          indexer_config=ZendeskIndexerConfig(
              batch_size=2,
              item_type=["tickets"]
          ),
          downloader_config=ZendeskDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
          source_connection_config=ZendeskConnectionConfig(
              access_config=ZendeskAccessConfig(
                  api_token=os.getenv("ZENDESK_API_TOKEN")
              ),
              subdomain=os.getenv("ZENDESK_SUBDOMAIN"),
              email=os.getenv("ZENDESK_EMAIL")
          ),
          partitioner_config=PartitionerConfig(
              partition_by_api=True,
              api_key=os.getenv("UNSTRUCTURED_API_KEY"),
              partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
              additional_partition_args={
                  "split_pdf_page": True,
                  "split_pdf_allow_failed": True,
                  "split_pdf_concurrency_level": 15
              }
          ),
          chunker_config=ChunkerConfig(chunking_strategy="by_title"),
          embedder_config=EmbedderConfig(embedding_provider="huggingface"),
          uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
      ).run()
  ```
</CodeGroup>

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:

* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`

* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  * `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
  * `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
  * The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.

  <Note>
    You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.

    The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

    If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).

    If you are using a **Business** account, the process
    for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
    For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
  </Note>


# Supported file types
Source: https://docs.unstructured.io/open-source/ingestion/supported-file-types



The Unstructured Ingest CLI and Unstructured Ingest Python library support processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.abw`         |
| `.bmp`         |
| `.csv`         |
| `.cwk`         |
| `.dbf`         |
| `.dif`\*       |
| `.doc`         |
| `.docx`        |
| `.dot`         |
| `.dotm`        |
| `.eml`         |
| `.epub`        |
| `.et`          |
| `.eth`         |
| `.fods`        |
| `.heic`        |
| `.htm`         |
| `.html`        |
| `.hwp`         |
| `.jpeg`        |
| `.jpg`         |
| `.md`          |
| `.mcw`         |
| `.msg`         |
| `.mw`          |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pbd`         |
| `.pdf`         |
| `.png`         |
| `.pot`         |
| `.ppt`         |
| `.pptm`        |
| `.pptx`        |
| `.prn`         |
| `.rst`         |
| `.rtf`         |
| `.sdp`         |
| `.sxg`         |
| `.tif`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |
| `.zabw`        |

By file type:

| Category          | File types                                                        |
| ----------------- | ----------------------------------------------------------------- |
| Apple             | `.cwk`, `.mcw`                                                    |
| CSV               | `.csv`                                                            |
| Data Interchange  | `.dif`\*                                                          |
| dBase             | `.dbf`                                                            |
| E-mail            | `.eml`, `.msg`, `.p7s`                                            |
| EPUB              | `.epub`                                                           |
| HTML              | `.htm`, `.html`                                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.jpg`, `.png`, `.prn`, `.tif`, `.tiff` |
| Markdown          | `.md`                                                             |
| OpenOffice        | `.odt`                                                            |
| Org Mode          | `.org`                                                            |
| Other             | `.eth`, `.pbd`, `.sdp`                                            |
| PDF               | `.pdf`                                                            |
| Plain text        | `.txt`                                                            |
| PowerPoint        | `.pot`, `.ppt`, `.pptm`, `.pptx`                                  |
| reStructured Text | `.rst`                                                            |
| Rich Text         | `.rtf`                                                            |
| Spreadsheet       | `.et`, `.fods`, `.mw`, `.xls`, `.xlsx`                            |
| StarOffice        | `.sxg`                                                            |
| TSV               | `.tsv`                                                            |
| Word processing   | `.abw`, `.doc`, `.docx`, `.dot`, `.dotm`, `.hwp`, `.zabw`         |
| XML               | `.xml`                                                            |

`*` For `.dif`, `\n` characters in `.dif` files are supported, but `\r\n` characters will raise the error
`UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type`.


# Docker installation
Source: https://docs.unstructured.io/open-source/installation/docker-installation



Follow these steps to run the Unstructured open source library inside a Docker container.

<Steps>
  <Step title="Install and run Docker">
    If you do not have Docker already installed and running, you can install and run a tool such as Docker Desktop, which is
    available for macOS, Windows, and Linux. Learn how to install and run:

    * [Docker Desktop on Mac](https://docs.docker.com/desktop/setup/install/mac-install/)
    * [Docker Desktop on Windows](https://docs.docker.com/desktop/setup/install/windows-install/)
    * [Docker Desktop on Linux](https://docs.docker.com/desktop/setup/install/linux/)
  </Step>

  <Step title="Pull the Unstructured Docker image">
    <Info>If you are an experienced Docker user, you plan to parse only a single type of data, and you want to accelerate the image-building process, you can [build your own Docker image](#building-your-own-docker-image) instead of pulling the latest prebuilt image.</Info>

    <Tabs>
      <Tab title="Docker Desktop UI">
        <Note>
          The following steps are for AMD64-based systems.

          If you are using an ARM64-based system (such as Apple Silicon), follow the instructions on the **Docker CLI** tab in this step instead.
        </Note>

        1. In your Docker Desktop UI's search box, enter `downloads.unstructured.io/unstructured-io/unstructured:latest`.
        2. On the **Images** tab, next to **unstructured-io/unstructured**, click **Pull**.

        To list the available images on your machine, in the sidebar, click **Images**.

        To remove this image from your machine at any time, click the trash can (**Delete**) icon next to the image in the
        list of available images.
      </Tab>

      <Tab title="Docker CLI">
        From your terminal or command prompt, run the following command.

        <Tip>If you have the Docker Desktop UI running, you can click the **Terminal** button in the UI's lower right corner to run a Docker CLI session from within the Docker Desktop UI.</Tip>

        For AMD64-based systems, run the following command:

        ```bash theme={null}
        # The AMD64 platform is the default.
        docker pull downloads.unstructured.io/unstructured-io/unstructured:latest

        # Or, to explicitly specify the AMD64 platform:
        docker pull --platform=linux/amd64 downloads.unstructured.io/unstructured-io/unstructured:latest
        ```

        For ARM64-based systems (such as Apple Silicon), run the following command instead:

        ```bash theme={null}
        docker pull --platform=linux/arm64 downloads.unstructured.io/unstructured-io/unstructured:latest
        ```

        To list the available images on your machine, run the following command:

        ```bash theme={null}
        docker images
        ```

        To remove this image from your machine at any time, run the following command:

        ```bash theme={null}
        docker rmi downloads.unstructured.io/unstructured-io/unstructured:latest
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Create and run a container from the image">
    <Tabs>
      <Tab title="Docker Desktop UI">
        <Note>
          The following steps are for AMD64-based systems.

          If you are using an ARM64-based system (such as Apple Silicon), follow the instructions on the **Docker CLI** tab in this step instead.
        </Note>

        1. In the Docker Desktop UI's sidebar, click **Images**.
        2. Next to **unstructured-io/unstructured**, click the play (**Run**) icon.
        3. Expand **Optional settings**.
        4. For **Container name**, enter some name for your container, such as `unstructured`.
        5. In the sidebar, click **Containers**.
        6. Next to your container, click the play (**Start**) icon.
      </Tab>

      <Tab title="Docker CLI">
        For AMD64-based systems, run the following command, replacing `<container-name>` with some name for your container, such as `unstructured`:

        ```bash theme={null}
        # The AMD64 platform is the default.
        docker run -dt --name <container-name> downloads.unstructured.io/unstructured-io/unstructured:latest

        # Or, to explicitly specify the AMD64 platform:
        docker run -dt --platform=linux/amd64 --name <container-name> downloads.unstructured.io/unstructured-io/unstructured:latest
        ```

        For ARM64-based systems (such as Apple Silicon), run the following command instead, replacing `<container-name>` with some name for your container, such as `unstructured`:

        ```bash theme={null}
        docker run -dt --platform=linux/arm64 --name <container-name> downloads.unstructured.io/unstructured-io/unstructured:latest
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Interact with the Unstructured open source library by running code inside the container">
    <Tabs>
      <Tab title="Docker Desktop UI">
        1. In the Docker Desktop UI, in the lower right corner, click the **Terminal** button.

        2. To start a terminal session inside the container, run the following command, replacing `<container-name>` with the name of your container, such as `unstructured`:

           ```bash theme={null}
           docker exec -it <container-name> bash
           ```

        3. Run Unstructured open source library calls from inside the container. For example, start the Python interpreter:

           ```bash theme={null}
           python
           ```

           And then run the following commands, one command at a time, to make calls to the Unstructured open source library.
           These calls process a PDF file in the `/app/example-docs/pdf` directory named `layout-parser-paper.pdf`. The
           processed data is written as a JSON file named `layout-parser-paper-output.json` in that same directory:

           ```bash theme={null}
           >>> from unstructured.partition.pdf import partition_pdf
           >>> from unstructured.staging.base import elements_to_json
           >>> elements = partition_pdf(filename="/app/example-docs/pdf/layout-parser-paper.pdf")
           >>> elements_to_json(elements=elements, filename="/app/example-docs/pdf/layout-parser-paper-output.json")
           ```

           After the last call finishes running, exit the Python interpreter, and then print the contents of the JSON file to the terminal:

           ```bash theme={null}
           >>> exit()

           cat ./example-docs/pdf/layout-parser-paper-output.json
           ```

        4. To exit the terminal session, run the following command, or press `Ctrl+D`:

           ```bash theme={null}
           exit
           ```
      </Tab>

      <Tab title="Docker CLI">
        1. Run the following command, replacing `<container-name>` with the name of your container, such as `unstructured`:

           ```bash theme={null}
           docker exec -it <container-name> bash
           ```

        2. Run Unstructured open source library calls from inside the container. For example, start the Python interpreter:

           ```bash theme={null}
           python
           ```

           And then run the following commands, one command at a time, to make calls to the Unstructured open source library.
           These calls process a PDF file in the `/app/example-docs/pdf` directory named `layout-parser-paper.pdf`. The
           processed data is written as a JSON file named `layout-parser-paper-output.json` in that same directory:

           ```bash theme={null}
           >>> from unstructured.partition.pdf import partition_pdf
           >>> from unstructured.staging.base import elements_to_json
           >>> elements = partition_pdf(filename="/app/example-docs/pdf/layout-parser-paper.pdf")
           >>> elements_to_json(elements=elements, filename="/app/example-docs/pdf/layout-parser-paper-output.json")
           ```

           After the last call finishes running, exit the Python interpreter, and then print the contents of the JSON file to the terminal:

           ```bash theme={null}
           >>> exit()

           cat ./example-docs/pdf/layout-parser-paper-output.json
           ```

        3. To exit the terminal session, run the following command, or press `Ctrl+D`:

           ```bash theme={null}
           exit
           ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Interact with the Unstructured open source library by running code outside the container">
    You can also interact with the Unstructured open source library by running code that is on the
    same machine as the running container but not within the container itself. To do this, you can
    use the Docker CLI to create a container that mounts the local directory containing the
    code into the container itself, and then run that code from the container.

    1. Run one of the following commands, replacing the following placeholders with the appropriate values:

       * Replace `<host-path>` with the path to the directory containing your code, for example `/Users/<username>/my_example_code/`.
       * Replace `<container-path>` with the path to some directory within the container to mount `<host-path>` into, for example `/app/my_example_code/`. If
         `<container-path>` does not already exist, it will be created at the same time that the container is created.
       * Replace `<container-name>` with some name for your container, such as `unstructured_mount`.

       For AMD64-based systems, run the following command:

       ```bash theme={null}
       # The AMD64 platform is the default.
       docker run -dt -v <host-path>:<container-path>--name <container-name> downloads.unstructured.io/unstructured-io/unstructured:latest

       # Or, to explicitly specify the AMD64 platform:
       docker run -dt -v <host-path>:<container-path> --platform=linux/amd64 --name <container-name> downloads.unstructured.io/unstructured-io/unstructured:latest
       ```

       For ARM64-based systems (such as Apple Silicon), run the following command instead:

       ```bash theme={null}
       docker run -dt -v <host-path>:<container-path> --platform=linux/arm64 --name <container-name> downloads.unstructured.io/unstructured-io/unstructured:latest
       ```

    2. Start a terminal session inside the container by running the following command, replacing `<container-name>` with the name of your container, such as `unstructured_mount`:

       ```bash theme={null}
       docker exec -it <container-name> bash
       ```

    3. Add `<container-path>` to the `PYTHONPATH` environment variable within the container by running the following commands,
       replacing `<container-path>` with the path to the target directory within the container:

       ```bash theme={null}
       PYTHONPATH="${PYTHONPATH}:<container-path>"
       export PYTHONPATH
       ```

    4. Run Unstructured open source library calls, referencing your code from `<container-path>`.

       For example, if you have a file named `main.py` in `<host-path>`that contains the four commands following `>>>` from the previous step,
       you can run it as follows, replacing `<container-path>` with the path to the target directory within the container:

       ```bash theme={null}
       python <container-path>/main.py
       ```

       To print the contents of the JSON file to the terminal, run the following command:

       ```bash theme={null}
       cat /app/example-docs/pdf/layout-parser-paper-output.json
       ```

    5. To exit the terminal session, run the following command, or press `Ctrl+D`:

       ```bash theme={null}
       exit
       ```
  </Step>

  <Step title="Stop running the container">
    If you do not need the keep running the container, you can stop it as follows:

    <Tabs>
      <Tab title="Docker Desktop UI">
        1. In the Docker Desktop UI, in the sidebar, click **Containers**.
        2. Next to your container, click the square (**Stop**) icon.
      </Tab>

      <Tab title="Docker CLI">
        Run the following command, replacing `<container-name>` with the name of your container, such as `unstructured` or `unstructured_mount`:

        ```bash theme={null}
        docker stop <container-name>
        ```
      </Tab>
    </Tabs>
  </Step>
</Steps>

## Building your own Docker image

You can build your own Docker image instead of pulling the latest prebuilt image.
If you only plan to parse a single type of data, you can accelerate the
build process by excluding certain packages or requirements needed for other data types. Refer to the
[Dockerfile](https://github.com/Unstructured-IO/unstructured/blob/main/Dockerfile) to determine which lines
are necessary for your requirements.

```bash theme={null}
make docker-build

# Start a Bash shell inside of the running Docker container.
make docker-start-bash
```


# Full installation
Source: https://docs.unstructured.io/open-source/installation/full-installation



To install the Unstructured open source library on a local development machine, run one or more of the following commands.

These commands assume that you are using the Python package and project manager
[uv](https://docs.astral.sh/uv/), running within an activated [venv](https://docs.python.org/3/library/venv.html)
virtual environment that was created with `uv`. However, `uv` and `venv` are not required.

To work with all [supported file types](/open-source/introduction/supported-file-types), run:

```bash theme={null}
uv add "unstructured[all-docs]"
```

To conserve disk space and reduce code dependencies, you can run the following command instead to work with a
default set of supported file types:

```bash theme={null}
uv add unstructured
```

The preceding command supports plain text files (`.txt`), HTML files (`.html`), XML files (`.xml`), and emails (`.eml`, `.msg`, and `.p7s`) by default.

To further conserve disk space and reduce code dependencies, you can run the following command instead, replacing `<extra>` with the appropriate extra for the target file type:

```bash theme={null}
uv add "unstructured[<extra>]"
```

The following file type extras are available:

* `all-docs` (for all supported file types in this list)
* `csv` (for `.csv` files only)
* `docx` (for `.doc` and `.docx` files only)
* `epub` (for `.epub` files only)
* `image` (for all supported image file types: `.bmp`, `.heic`, `.jpeg`, `.png`, and `.tiff`)
* `md` (for `.md` files only)
* `odt` (for `.odt` files only)
* `org` (for `.org` files only)
* `pdf` (for `.pdf` files only)
* `pptx` (for `.ppt` and `.pptx` files only)
* `rst` (for `.rst` files only)
* `rtf` (for `.rtf` files only)
* `tsv` (for `.tsv` files only)
* `xlsx` (for `.xls` and `.xlsx` files only)

Note that you can install multiple extras at the same time by separating them with commas, for example:

```bash theme={null}
uv add "unstructured[pdf,docx]"
```

For maximum compatiblity, you should also install the following system dependencies:

* [libmagic-dev](https://man7.org/linux/man-pages/man3/libmagic.3.html) (for filetype detection)
* [poppler-utils](https://poppler.freedesktop.org/) and [tesseract-ocr](https://github.com/tesseract-ocr/tesseract) (for images and PDFs), and `tesseract-lang` (for additional language support)
* [libreoffice](https://www.libreoffice.org/discover/libreoffice/) (for Microsoft Office documents)
* [pandoc](https://pandoc.org/) (for `.epub`, `.odt`, and `.rtf` files. For `.rtf` files, you must have version 2.14.2 or newer. Running [this script](https://github.com/Unstructured-IO/unstructured/blob/main/scripts/install-pandoc.sh) will install the correct version for you.)

Installation instructured for these system dependencies vary by operating system type. For details, follow the preceding links or see your
operating system's documentation.


# Overview
Source: https://docs.unstructured.io/open-source/installation/overview

Unstructured open source library offers flexible options for installation.

Refer to the Full installation guide to learn how to install Unstructured with all modules, or how to trim down the installation
to only what you require.

Refer to the Docker installation to learn how to run Unstructured in a Docker container.

<CardGroup>
  <Card title="Full Installation" icon="download" href="/open-source/installation/full-installation" />

  <Card title="Docker Installation" icon="docker" href="/open-source/installation/docker-installation" />
</CardGroup>


# Integrations
Source: https://docs.unstructured.io/open-source/integrations

Integrate your model development pipeline with your favorite machine learning frameworks and libraries, and prepare your data for ingestion into downstream systems. Most of our integrations come in the form of [staging functions](/open-source/core-functionality/staging), which take a list of `Element` objects as input and return formatted dictionaries as output.

## `Integration with Argilla`

You can convert a list of `Text` elements to an [Argilla](https://www.argilla.io/) `Dataset` using the [stage\_for\_argilla](/open-source/core-functionality/staging#stage-for-argilla) staging function. Specify the type of dataset to be generated using the `argilla_task` parameter. Valid values are `"text_classification"`, `"token_classification"`, and `"text2text"`. Follow the link for more details on usage.

## `Integration with Baseplate`

[Baseplate](https://docs.baseplate.ai/introduction) is a backend optimized for use with LLMs that has an easy to use spreadsheet interface. The `unstructured` library offers a staging function to convert a list of `Element` objects into the [rows format](https://docs.baseplate.ai/api-reference/documents/overview) required by the Baseplate API. See the [stage\_for\_baseplate](/open-source/core-functionality/staging#stage-for-baseplate) documentation for information on how to stage elements for ingestion into Baseplate.

## `Integration with Datasaur`

You can format a list of `Text` elements as input to token based tasks in [Datasaur](https://datasaur.ai/) using the [stage\_for\_datasaur](/open-source/core-functionality/staging#stage-for-datasaur) staging function. You will obtain a list of dictionaries indexed by the keys `"text"` with the content of the element, and `"entities"` with an empty list. Follow the link to learn how to customise your entities and for more details on usage.

## `Integration with Hugging Face`

You can prepare `Text` elements for processing in Hugging Face [Transformers](https://huggingface.co/docs/transformers/index) pipelines by splitting the elements into chunks that fit into the model’s attention window using the [stage\_for\_transformers](/open-source/core-functionality/staging#stage-for-transformers) staging function. You can customise the transformation by defining the `buffer` and `window_size`, the `split_function` and the `chunk_separator`. if you need to operate on text directly instead of `unstructured` `Text` objects, use the [chunk\_by\_attention\_window](/open-source/core-functionality/staging#stage-for-transformers) helper function. Follow the links for more details on usage.

## `Integration with Labelbox`

You can format your outputs for use with [LabelBox](https://labelbox.com/) using the [stage\_for\_label\_box](/open-source/core-functionality/staging#stage-for-label-box) staging function. LabelBox accepts cloud-hosted data and does not support importing text directly. With this integration you can stage the data files in the `output_directory` to be uploaded to a cloud storage service (such as S3 buckets) and get a config of type `List[Dict[str, Any]]` that can be written to a `.json` file and imported into LabelBox. Follow the link to see how to generate the `config.json` file that can be used with LabelBox, how to upload the staged data files to an S3 bucket, and for more details on usage.

## `Integration with Label Studio`

You can format your outputs for upload to [Label Studio](https://labelstud.io/) using the [stage\_for\_label\_studio](/open-source/core-functionality/staging#stage-for-label-studio) staging function. After running `stage_for_label_studio`, you can write the results to a JSON folder that is ready to be included in a new Label Studio project. You can also include pre-annotations and predictions as part of your upload.

See the [Label Studio docs](https://labelstud.io/tags/labels.html) for a full list of options for labels and annotations.

## `Integration with LangChain`

Our integration with [LangChain](https://github.com/hwchase17/langchain) makes it incredibly easy to combine language models with your data, no matter what form it is in. The [Unstructured.io File Loader](https://python.langchain.com/en/latest/modules/indexes/document_loaders/examples/unstructured_file.html) extracts the text from a variety of unstructured text files using our `unstructured` library. It is designed to be used as a way to load data into LangChain. Here is the simplest way to use the `UnstructuredFileLoader` in `langchain`.

```python theme={null}
from langchain.document_loaders import UnstructuredFileLoader

loader = UnstructuredFileLoader("state_of_the_union.txt")
loader.load()

```

Checkout the [LangChain docs](https://python.langchain.com/en/latest/modules/indexes/document_loaders.html) for more examples about how to use Unstructured data loaders.

## `Integration with LlamaIndex`

To use `Unstructured.io File Loader` you will need to have [LlamaIndex](https://github.com/jerryjliu/llama_index) 🦙 (GPT Index) installed in your environment. Just `pip install llama-index` and then pass in a `Path` to a local file. Optionally, you may specify split\_documents if you want each element generated by `unstructured` to be placed in a separate document. Here is a simple example of how to use it:

```python theme={null}
from pathlib import Path
from llama_index import download_loader


UnstructuredReader = download_loader("UnstructuredReader")

loader = UnstructuredReader()
documents = loader.load_data(file=Path('./10k_filing.html'))

```

See [here](https://llamahub.ai/) for more LlamaHub examples.

## `Integration with Pandas`

You can convert a list of `Element` objects to a Pandas dataframe with columns for the text from each element and their types such as `NarrativeText` or `Title` using the [convert\_to\_dataframe](/open-source/core-functionality/staging#convert-to-dataframe) staging function. Follow the link for more details on usage.

## `Integration with Prodigy`

You can format your JSON or CSV outputs for use with [Prodigy](https://prodi.gy/docs/api-loaders) using the [stage\_for\_prodigy](/open-source/core-functionality/staging#stage-for-prodigy) and [stage\_csv\_for\_prodigy](/open-source/core-functionality/staging#stage-csv-for-prodigy) staging functions. After running `stage_for_prodigy` | `stage_csv_for_prodigy`, you can write the results to a `.json` | `.jsonl` or a `.csv` file that is ready to be used with Prodigy. Follow the links for more details on usage.

## `Integration with Weaviate`

[Weaviate](https://weaviate.io/) is an open-source vector database that allows you to store data objects and vector embeddings from a variety of ML models. Storing text and embeddings in a vector database such as Weaviate is a key component of the [emerging LLM tech stack](https://medium.com/@unstructured-io/llms-and-the-emerging-ml-tech-stack-bdb189c8be5c). See the [stage\_for\_weaviate](open-source/core-functionality/staging#stage-for-weaviate) docs for details on how to upload `unstructured` outputs to Weaviate. An example notebook is also available [here](https://github.com/Unstructured-IO/unstructured/tree/main/examples/weaviate).


# Overview
Source: https://docs.unstructured.io/open-source/introduction/overview



<Note>The Unstructured open source library is designed as a starting point for quick prototyping and has [limits](#limits). For production scenarios, use the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview) instead.</Note>

<Tip>To start using the Unstructured open source library right away, skip ahead to the [quickstart](/open-source/introduction/quick-start).</Tip>

The Unstructured open source library ([GitHub](https://github.com/Unstructured-IO/unstructured), [PyPI](https://pypi.org/project/unstructured/)) offers an open-source toolkit
designed to simplify the ingestion and pre-processing of diverse data formats, including images and text-based documents
such as PDFs, HTML files, Word documents, and more. With a focus on optimizing data workflows for Large Language Models (LLMs),
the Unstructured open source library provides modular functions and connectors that work seamlessly together. This cohesive system ensures
efficient transformation of unstructured data into structured formats, while also offering adaptability to various platforms
and use cases.

## Key functionality

* **Precise document extraction**: Unstructured offers advanced capabilities in extracting elements and metadata from documents. This includes a variety of document element types and metadata. Learn more about [Document elements and metadata](../concepts/document-elements).

* **Robust file support**: The platform supports a wide array of file types, ensuring versatility in handling different document formats from PDF, Images, HTML, and many more. Detailed information on supported file types can be found [here](/open-source/introduction/supported-file-types).

* **Robust core functionality**: Unstructured provides a suite of core functionalities critical for efficient data processing. This includes:

  * [Partitioning](/open-source/core-functionality/partitioning): The partitioning functions in Unstructured enable the extraction of structured content from raw, unstructured documents. This feature is crucial for transforming unorganized data into usable formats, aiding in efficient data processing and analysis.

  * [Cleaning](/open-source/core-functionality/cleaning): Data preparation for NLP models often requires cleaning to ensure quality. The Unstructured library includes cleaning functions that assist in sanitizing output, removing unwanted content, and improving the performance of NLP models. This step is essential for maintaining the integrity of data before it is passed to downstream applications.

  * [Extracting](/open-source/core-functionality/extracting): This functionality allows for the extraction of specific entities within documents. It is designed to identify and isolate relevant pieces of information, making it easier for users to focus on the most pertinent data in their documents.

  * [Staging](/open-source/core-functionality/staging): Staging functions help prepare your data for ingestion into downstream systems. Please note that this functionality is being deprecated in favor of [destination connectors](/open-source/ingestion/destination-connectors/overview) in the [Unstructured Ingest CLI and Unstructured Ingest Python library](/open-source/ingestion/overview).

  * [Chunking](/open-source/core-functionality/chunking): The chunking process in Unstructured is distinct from conventional methods. Instead of relying solely on text-based features to form chunks, Unstructured uses a deep understanding of document formats to partition documents into semantic units (document elements).

## Common use cases

* Pretraining models
* Fine-tuning models
* Retrieval-augmented generation (RAG)
* Traditional ETL

<Note>GPU usage is not supported for the Unstructured open source library.</Note>

## Limits

The Unstructured open source library has the following limits as compared to the [Unstructured Pipelines](/pipelines/overview) and the [Unstructured API](/api-reference/overview):

* Not designed for production scenarios.
* Significantly decreased performance on document and table extraction.
* No access to Unstructured's latest vision language model (VLM) offerings.
* No access to Unstructured's fine-tuned OCR models.
* No access to Unstructured's by-page and by-similarity chunking strategies.
* No support for generating embeddings in the core [Unstructured](https://github.com/Unstructured-IO/unstructured) open source offering. (However, you can
  generate embeddings as a separate step manually. [Learn how](/open-source/core-functionality/embedding). Also, there is built-in support for generating embeddings by using the open source's
  [Unstructured Ingest CLI](/open-source/ingestion/ingest-cli) and [Unstructured Ingest Python library](/open-source/ingestion/python-ingest) offerings.
  [Learn more](/open-source/how-to/embedding).)
* No support for Unstructured's enrichment types such as image descriptions, table descriptions, and named entity recognition (NER).
* Lack of support for SOC 2 Type 1, SOC 2 Type 2, HIPAA, GDPR, ISO 27001, FedRAMP, and CMMC 2.0 Level 2 compliance.
* No authentication or identity management in the core open source offering for local document processing.
* No incremental data loading.
* No ETL job scheduling or monitoring.
* No image extraction from documents.
* Less sophisticated document hierarchy detection.
* You must manage many of your own code dependencies, for instance for libraries such as Poppler and Tesseract.
* For local document processing, you must manage your own infrastructure.

The following table compares the [Unstructured Pipelines](/pipelines/overview) and the
[Unstructured API](/api-reference/overview) to the [Unstructured open source library](/open-source/introduction/overview).

|                                                                                           | Unstructured Pipelines/API | Unstructured open source library |
| ----------------------------------------------------------------------------------------- | -------------------------- | -------------------------------- |
| *Use for...*                                                                              | *Production scenarios*     | *Quick prototyping*              |
| *Deployment options*                                                                      |                            |                                  |
| Software as a service (Saas)                                                              | <span>Yes</span>           | <span>No</span>                  |
| In-VPC                                                                                    | <span>Yes</span>           | <span>Yes</span>                 |
| *Security and compliance*                                                                 |                            |                                  |
| One-click, single sign-on (SSO) authentication                                            | <span>Yes</span>           | <span>No</span>                  |
| SOC 2 Type 2, HIPAA, GDPR, ISO 27001, FedRAMP, and CMMC 2.0 Level 2 compliance            | <span>Yes</span>           | <span>No</span>                  |
| *Connectivity*                                                                            |                            |                                  |
| 20+ source connectors                                                                     | <span>Yes</span>           | <span>Yes</span>                 |
| 20+ destination connectors                                                                | <span>Yes</span>           | <span>Yes</span>                 |
| 25+ supported source file types                                                           | <span>Yes</span>           | <span>Yes</span>                 |
| 20+ additional supported source file types (45+ total)                                    | <span>Yes</span>           | <span>No</span>                  |
| 35+ types of metadata                                                                     | <span>Yes</span>           | <span>Yes</span>                 |
| Built-in integrations with top VLM, LLM, and embedding model providers                    | <span>Yes</span>           | <span>No</span>                  |
| *Document processing*                                                                     |                            |                                  |
| Transforms source documents into Unstructured's canonical JSON schema                     | <span>Yes</span>           | <span>Yes</span>                 |
| Smart routing logic for the most efficient and cost-effective document processing         | <span>Yes</span>           | <span>No</span>                  |
| Advanced semantic chunking strategies, summary generation, and structured data generation | <span>Yes</span>           | <span>No</span>                  |
| Automatic embedding and chunking logic selection                                          | <span>Yes</span>           | <span>No</span>                  |
| *Performance*                                                                             |                            |                                  |
| Industry-leading file transformation and table extraction logic                           | <span>Yes</span>           | <span>No</span>                  |
| 50x transformation speeds with multi-node serving and auto-scaling                        | <span>Yes</span>           | <span>No</span>                  |
| *Usage and billing*                                                                       |                            |                                  |
| Real-time billing and usage dashboard                                                     | <span>Yes</span>           | <span>No</span>                  |
| User interface and API                                                                    | <span>Yes</span>           | <span>No</span>                  |
| Use with no-code UI or API                                                                | <span>Yes</span>           | <span>No</span>                  |
| *Additional features*                                                                     |                            |                                  |
| Workflow management and scheduling                                                        | <span>Yes</span>           | <span>No</span>                  |
| More frequent updates                                                                     | <span>Yes</span>           | <span>No</span>                  |

For additional considerations, see [When Open Source Isn't Good Enough](https://unstructured.io/blog/when-open-source-isnt-good-enough).

See also the following 26-minute video:

<iframe title="YouTube video player" />

## Pricing

Calls to the Unstructured open source library that are routed to Unstructured's software-as-a-service (SaaS)
for processing (for example, by calling the
[partition\_via\_api](/open-source/core-functionality/partitioning#partition-via-api) or
[partition\_multiple\_via\_api](/open-source/core-functionality/partitioning#partition-multiple-via-api) functions with an
Unstructured API key and an Unstructured SaaS URL) require an Unstructured account for billing purposes.

Unstructured offers different account types with different pricing plans:

* <Icon icon="person" />  **Let's Go** and **Pay-As-You-Go** - A single user, with a single workspace, hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="building" />  **Business** - Multiple users and workspaces, with three options:

  * <Icon icon="people-group" />  **Business SaaS** - Hosted alongside other accounts on Unstructured's cloud infrastructure.
  * <Icon icon="shield-halved" />  **Dedicated instance** - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control.
  * <Icon icon="shield" />  **In-VPC** - Hosted within your own VPC on your own cloud infrastructure.

  **Business** accounts also allow for robust customization of Unstructured's features for your unique needs.

For more details, see the [Unstructured Pricing](https://unstructured.io/pricing) page.

To upgrade your account from **Let's Go** or **Pay-As-You-Go** to **Business**,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Some of these plans have billing details that are determined on a per-page basis.

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

## <Icon icon="question" />    Questions? Need help?

* For questions and technical support for the Unstructured open source library, use our [Slack community](https://short.unstructured.io/pzw05l7).
* If you need help with your Unstructured account, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# Quickstart
Source: https://docs.unstructured.io/open-source/introduction/quick-start



<Note>This quickstart uses the Unstructured open source library, which is designed as a starting point for quick prototyping and has [limits](#limits). For production scenarios, use the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview) instead.</Note>

In this quickstart, you use the [Unstructured open source library](/open-source/introduction/overview)
([GitHub](https://github.com/Unstructured-IO/unstructured),
[PyPI](https://pypi.org/project/unstructured/)) along with Python on your local development machine to partition a PDF file into a standard set of
[Unstructured document elements and metadata](/open-source/concepts/document-elements). You can use these elements and
metadata as input into your RAG applications, AI agents, model fine-tuning tasks, and more.

<Steps>
  <Step title="Prerequisites">
    To complete this quickstart, you need:

    * A Python virtual environment manager is recommended to manage your Python code dependencies.
      This quickstart uses [uv](https://docs.astral.sh/uv/) for managing virtual environments and
      [venv](https://docs.python.org/3/library/venv.html) as the virtual environment type. Installation and
      use of `uv` and `venv` are described in the following steps.
      However, `uv` and `venv` are not required to use the Unstructured open source library.
    * Python 3.9 or higher. You can use `uv` to install Python if needed, as described in the following steps.
    * A PDF file on your local machine. If you do not have a PDF file available, this quickstart provides a sample PDF file named
      `layout-parser-paper.pdf` that you can download in a later step. (The Unstructured open source library provides
      [support for additional file types](/open-source/introduction/supported-file-types) as well.)
  </Step>

  <Step title="Install uv">
    <Tabs>
      <Tab title="macOS, Linux">
        To use `curl` with `sh`:

        ```bash theme={null}
        curl -fsSL https://get.uv.dev | bash
        ```

        To use `wget` with `sh` instead:

        ```bash theme={null}
        wget -qO- https://astral.sh/uv/install.sh | sh
        ```
      </Tab>

      <Tab title="Windows">
        To use PowerShell with `irm` to download the script and run it with `iex`:

        ```powershell theme={null}
        powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
        ```
      </Tab>
    </Tabs>

    To install `uv` by using other approaches such as PyPI, Homebrew, or WinGet,
    see [Installing uv](https://docs.astral.sh/uv/getting-started/installation/).
  </Step>

  <Step title="Install Python">
    `uv` will detect and use Python if you already have it installed.
    To view a list of installed Python versions, run the following command:

    ```bash theme={null}
    uv python list
    ```

    If, however, you do not already have Python installed, you can install a version of Python for use with `uv`
    by running the following command. For example, this command installs Python 3.12 for use with `uv`:

    ```bash theme={null}
    uv python install 3.12
    ```
  </Step>

  <Step title="Create a uv project">
    Use `uv` to create a project by switching to the directory on your development machine where you want to
    create the project and then running the following command:

    ```bash theme={null}
    uv init
    ```
  </Step>

  <Step title="Create a venv virtual environment">
    To isolate and manage your project's code dependencies, from your project directory, use `uv` to create a virtual environment with `venv` by running the following command:

    ```bash theme={null}
    # Create the virtual environment by using the current Python version:
    uv venv

    # Or, if you want to use a specific Python version:
    uv venv --python 3.12
    ```
  </Step>

  <Step title="Activate the virtual environment">
    To activate the `venv` virtual environment, run one of the following commands:

    <Tabs>
      <Tab title="macOS, Linux">
        * For `bash` or `zsh`, run `source .venv/bin/activate`
        * For `fish`, run `source .venv/bin/activate.fish`
        * For `csh` or `tcsh`, run `source .venv/bin/activate.csh`
        * For `pwsh`, run `.venv/bin/Activate.ps1`
      </Tab>

      <Tab title="Windows">
        * For `cmd.exe`, run `.venv\Scripts\activate.bat`
        * For `PowerShell`, run `.venv\Scripts\Activate.ps1`
      </Tab>
    </Tabs>

    To deactivate the virtual environment at any time, run `deactivate`.
  </Step>

  <Step title="Install the Unstructured open source library">
    With the virtual environment activated to enable code dependency isolation and management, use `uv` to install the Unstructured open source library by running the following command:

    ```bash theme={null}
    uv add unstructured
    ```

    The preceding command supports plain text files (`.txt`), HTML files (`.html`), XML files (`.xml`), and emails (`.eml`, `.msg`, and `.p7s`) without any additional dependencies.

    To work with other file types, you must also install these dependencies, as follows, replacing `<extra>` with the appropriate extra for the target file type:

    ```bash theme={null}
    uv add "unstructured[<extra>]"
    ```

    The following file type extras are available:

    * `all-docs` (for all supported file types in this list)
    * `csv` (for `.csv` files only)
    * `docx` (for `.doc` and `.docx` files only)
    * `epub` (for `.epub` files only)
    * `image` (for all supported image file types: `.bmp`, `.heic`, `.jpeg`, `.png`, and `.tiff`)
    * `md` (for `.md` files only)
    * `odt` (for `.odt` files only)
    * `org` (for `.org` files only)
    * `pdf` (for `.pdf` files only)
    * `pptx` (for `.ppt` and `.pptx` files only)
    * `rst` (for `.rst` files only)
    * `rtf` (for `.rtf` files only)
    * `tsv` (for `.tsv` files only)
    * `xlsx` (for `.xls` and `.xlsx` files only)

    As this quickstart uses a sample PDF file, run the following command:

    ```bash theme={null}
    uv add "unstructured[pdf]"
    ```

    Note that you can install multiple extras at the same time by separating them with commas, for example:

    ```bash theme={null}
    uv add "unstructured[pdf,docx]"
    ```
  </Step>

  <Step title="Install system dependencies">
    You maximum compatibility, you should also install the following system dependencies:

    * [libmagic-dev](https://man7.org/linux/man-pages/man3/libmagic.3.html) (for filetype detection)
    * [poppler-utils](https://poppler.freedesktop.org/) and [tesseract-ocr](https://github.com/tesseract-ocr/tesseract) (for images and PDFs), and `tesseract-lang` (for additional language support)
    * [libreoffice](https://www.libreoffice.org/discover/libreoffice/) (for Microsoft Office documents)
    * [pandoc](https://pandoc.org/) (for `.epub`, `.odt`, and `.rtf` files. For `.rtf` files, you must have version 2.14.2 or newer. Running [this script](https://github.com/Unstructured-IO/unstructured/blob/main/scripts/install-pandoc.sh) will install the correct version for you.)

    Installation instructured for these system dependencies vary by operating system type. For details, follow the preceding links or see your
    operating system's documentation.
  </Step>

  <Step title="Download the sample PDF file">
    Download the sample PDF file named `layout-parser-paper.pdf` from the following location to your local development machine:

    [https://github.com/Unstructured-IO/unstructured/tree/main/example-docs/pdf](https://github.com/Unstructured-IO/unstructured/tree/main/example-docs/pdf)

    (You can also use any other PDF file that you want to work with instead of this sample file, if you prefer.)
  </Step>

  <Step title="Add the Python code">
    In the project's `main.py` file, add the following Python code, replacing `<path/to>` with the
    path to the `layout-parser-paper.pdf` file that you downloaded to your local development machine.

    (If you want to use a different PDF file, replace `layout-parser-paper` with the name of that PDF file instead.)

    ```python theme={null}
    from unstructured.partition.pdf import partition_pdf
    from unstructured.staging.base import elements_to_json

    file_path = "<path/to>"
    base_file_name = "layout-parser-paper"

    def main():
        elements = partition_pdf(filename=f"{file_path}/{base_file_name}.pdf")
        elements_to_json(elements=elements, filename=f"{file_path}/{base_file_name}-output.json")

    if __name__ == "__main__":
        main()
    ```
  </Step>

  <Step title="Run the Python code">
    Use `uv` to run the preceding Python code by running the following command:

    ```bash theme={null}
    uv run main.py
    ```

    It might take a few minutes for the command to finish.
  </Step>

  <Step title="View the output">
    After the command finishes running successfully, view the Unstructured elements and metadata that were generated by opening the `layout-parser-paper-output.json` file in your editor. This file will be in
    the location as the original `layout-parser-paper.pdf` file.

    (If you used a different PDF file, the output file will be named `<your-file-name>-output.json` instead.)
  </Step>
</Steps>

## Next steps

* Learn more about the [available partition functions](/open-source/core-functionality/partitioning) in addition to `partition_pdf` for converting other types of files into standard [Unstructured document elements and metadata](/open-source/concepts/document-elements).
* By default, the preceding example uses the `auto` partitioning strategy. Learn about other [available partitioning strategies](/open-source/concepts/partitioning-strategies) for fine-tuned approaches to converting different types of files into Unstructured document elements.
* Learn about [available chunking functions](/open-source/core-functionality/chunking) for splitting up the text in your document elements into manageable chunks as needed to fit into your models' limited context windows.
* Learn about [available cleaning functions](/open-source/core-functionality/cleaning) for cleaning up your document elements' data as needed.
* Learn about [available extraction functions](/open-source/core-functionality/extracting) for getting precise information out of your document elements as needed.
* Learn about how to [generate vector embeddings](/open-source/core-functionality/embedding) for the text in your document elements for use in RAG applications, AI agents, model fine-tuning tasks, and more.
* For an additional code example, see the [Unstructured Quick Tour](https://colab.research.google.com/github/Unstructured-IO/notebooks/blob/main/notebooks/Unstructured_Quick_Tour.ipynb) Google Colab notebook.
* The Unstructured open source library is also available as a [Docker container](/open-source/installation/docker-installation).
* The [Unstructured Ingest CLI and Unstructured Ingest Python library](/open-source/ingestion/overview) build upon the Unstructured open source library by providing additional functionality such as batch file processing,
  ingesting files from remote source locations and sending the processed files' data to remote destination locations, creating programmatic ETL pipelines, optionally processing files on Unstructured-hosted compute resource instead of locally for improved performance and quality on a pay-as-you-go basis, and more.
* The [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview) are superior to the Unstructured open source library, the
  Unstructured Ingest CLI, and the Unstructured Ingest Python library. Unstructured Pipelines and the API are designed for production scenarios, with significantly increased performance and quality,
  the latest OCR and vision language models, advanced chunking strategies, security compliance, multi-user account management, job scheduling and monitoring, self-hosted deployment options, and more on a pay-as-you-go or subscription basis.

## Need help?

* Join the [Unstructured Slack community](https://short.unstructured.io/pzw05l7) and post your
  questions in the **# ask-for-help-open-source-library** channel.
* Post your bug reports and feature requests in the [Unstructured open source library GitHub repository](https://github.com/Unstructured-IO/unstructured/issues). These bug reports and feature requests are evaluated and addressed\
  based on the interest and availability of the open source community.


# Supported file types
Source: https://docs.unstructured.io/open-source/introduction/supported-file-types



The Unstructured open source library supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.bmp`         |
| `.csv`         |
| `.doc`         |
| `.docx`        |
| `.eml`         |
| `.epub`        |
| `.heic`        |
| `.html`        |
| `.jpeg`        |
| `.png`         |
| `.md`          |
| `.msg`         |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pdf`         |
| `.png`         |
| `.ppt`         |
| `.pptx`        |
| `.rst`         |
| `.rtf`         |
| `.tif`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |

By file type:

| Category          | File types                                        |
| ----------------- | ------------------------------------------------- |
| CSV               | `.csv`                                            |
| E-mail            | `.eml`, `.msg`, `.p7s`                            |
| EPUB              | `.epub`                                           |
| Excel             | `.xls`, `.xlsx`                                   |
| HTML              | `.html`                                           |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.png`, `.tif`, `.tiff` |
| Markdown          | `.md`                                             |
| Org Mode          | `.org`                                            |
| Open Office       | `.odt`                                            |
| PDF               | `.pdf`                                            |
| Plain text        | `.txt`                                            |
| PowerPoint        | `.ppt`, `.pptx`                                   |
| reStructured Text | `.rst`                                            |
| Rich Text         | `.rtf`                                            |
| TSV               | `.tsv`                                            |
| Word              | `.doc`, `.docx`                                   |
| XML               | `.xml`                                            |

See also [Partitioning](/open-source/core-functionality/partitioning).


# Account API keys and API URLs
Source: https://docs.unstructured.io/pipelines/account/api-key-url



An API key is a unique string of characters used to authenticate calls to an application programming interface (API).
In Unstructured, API keys are used as a kind of password that allows code or scripts to call the
[Unstructured API](/api-reference/overview) to automate an Unstructured account on behalf of an associated account user
or other entity tied to the API key that is provided with each call. This API key also enables Unstructured to associate
usage and billing information with the related Unstructured account.

You need an API key whenever you want to call the Unstructured API's
[Unstructured API's workflow operations](/api-reference/workflow/overview) or legacy
[Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview).

You do not need an API key to use [Unstructured Pipelines](/pipelines/overview) to create or manage connectors, workflows,
or jobs in your Unstructured account, or to manage your Unstructured account.

<Note>
  For an Unstructured **Business** account, before you click **API Keys**, make sure you have selected the
  [organizational workspace](/pipelines/account/workspaces) you want to access API keys and API URLs for.
</Note>

## View available API keys

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **API Keys**.
3. Existing API keys are listed on the **API Keys** tab.

## Create a new API key

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **API Keys**.
3. On the **API Keys** tab, click **Generate New Key**.
4. Enter some descriptive name for the API key, and then click **Save**.

The new API key is created and appears in the list of available API keys.

The only setting you can change for this new API key is its [status](#enable-or-disable-an-existing-api-key). You cannot change its
name, value, or any other settings.

## Get an existing API key's value

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **API Keys**.
3. On the **API Keys** tab, in the list of available keys, click the **Copy** icon for the target key. The key's value is copied to your system's clipboard.

If you lose this API key's value, simply repeat this procedure.

If you are concerned that an unauthorized user or other entity might have this API key's value, you should [permanently delete](#permanently-delete-an-existing-api-key) this key.

## Enable or disable an existing API key

You might want to temporarily disable an existing API key that you suspect is causing some kind of issue or might be compromised in some way.
However, you might want the option to re-enable it after you solve your issues or determine that it was not compromised.

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **API Keys**.
3. On the **API Keys** tab, in the list of available keys, switch the **Status** toggle on or off for the target API key to enable or disable that key, respectively.

## Permanently delete an existing API key

<Warning>
  Deleting an existing API key is a permanent operation. Deleted API keys cannot be recovered.

  You cannot use a deleted API key to call the Unstructured APIs.
</Warning>

You might want to permanently delete an existing API key that you know is causing issues or is compromised.

If you have any code or scripts that rely on the old, existing API key that you want to permanently delete, you should rotate the key first.
To rotate an old, existing API key before you permanently delete it:

1. Create a new, replacement API key by following the steps in [Create a new API key](#create-a-new-api-key), earlier on this page.
2. Update your code or scripts that rely on the old, existing API key's value that you want to permanently delete with the new, replacement API key's value.
3. Delete the old, existing API key.

To permanently delete an existing API key:

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **API Keys**.
3. On the **API Keys** tab, in the list of available keys, click the **Options** vertical ellipsis for the target API key.
4. Click **Delete**.
5. Confirm that you want to permanently delete the API key by clicking **Continue**.

## View available API URLs

In some cases, you might need to specify a particular API URL to call one of the Unstructured API's available Endpoints. For more information, see the getting started instructions for the
Unstructured API's [Unstructured API's workflow operations](/api-reference/workflow/overview) or legacy
[Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview).

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click **API Keys**.
3. On the **API Keys** tab, the available API URLs are listed.


# Account billing
Source: https://docs.unstructured.io/pipelines/account/billing



## Add, view, or change payment details

<Note>
  The following information applies only to Unstructured **Pay-As-You-Go** accounts.

  To add, view, or change payment details for an Unstructured **Business** account, contact your
  Unstructured Account Executive (AE). If you do not know who your assigned AE is,
  email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Note>

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click your user icon, and then click **Account Settings**.
3. In the top navigation bar, click the **Billing** tab.
4. Click **Manage Payment**; follow the on-screen instructions to add, view, or change your payment details; and then click **Save card**.

## View billing details

<Note>
  The following information applies only to Unstructured **Pay-As-You-Go** accounts.

  To view billing details for an Unstructured **Business** account, contact your
  Unstructured Account Executive (AE). If you do not know who your assigned AE is,
  email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Note>

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click your user icon, and then click **Account Settings**.
3. In the top navigation bar, click the **Billing** tab. Your account's billing details are displayed.

## How is pay-per-page billing calculated?

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

The **Billing** tab shows your account's current billing rate per page.

To complete usage-related tasks in your account, see [Account usage](/pipelines/account/usage).


# Organizational accounts
Source: https://docs.unstructured.io/pipelines/account/organizations



<Note>
  The following information applies only to Unstructured **Business** accounts. To upgrade from an Unstructured **Let's Go** or **Pay-As-You-Go** account
  to a **Business** account, contact your Unstructured sales representative, or
  email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

An *organizational account* is an Unstructured account that allows
multiple users to access resources and share billing. An
organizational account contains a collection of Unstructured
[workspaces](/pipelines/account/workspaces) that can be shared with subsets
of users within the account.  A *workspace* is a collection of resources, such as connectors,
workflows, jobs, and API keys, within an organization. An
organizational account can have multiple workspaces.

#### Organizational accounts for Business SaaS customers

Business SaaS accounts support both *organizational accounts* and
*personal accounts*. A Business SaaS organizational account manages
access to the organization and its workspaces by inviting users with
Unstructured personal accounts. Any usage by invited users within the
organizational account is billed to that organizational account.
Personal accounts include a *personal workspace* that is separate from
the organization's workspaces and belongs to the individual user's
personal account.

## Organizational account architecture

Each organizational account operates independently of all other organizational accounts.

Each organizational account can have multiple workspaces. Each workspace operates independently of all other workspaces.

```mermaid theme={null}
graph LR
    Org1[Organization 1]
    Org2[Organization 2]
    Org3[Organization 3]
    WS1A[Workspace 1A]
    WS2A[Workspace 2A]
    WS2B[Workspace 2B]
    WS3A[Workspace 3A]
    WS3B[Workspace 3B]
    WS3C[Workspace 3C]

    Org1 --> WS1A
    
    Org2 --> WS2A
    Org2 --> WS2B

    Org3 --> WS3A
    Org3 --> WS3B
    Org3 --> WS3C
```

Each organizational account can have multiple users. Users can have access to multiple organizational accounts.

```mermaid theme={null}
graph LR
    User1[User 1]
    User2[User 2]
    User3[User 3]
    Org1[Organization 1]
    Org2[Organization 2]
    Org3[Organization 3]

    User1 -.-> Org1
    User2 -.-> Org1
    User2 -.-> Org2
    User3 -.-> Org1
    User3 -.-> Org2
    User3 -.-> Org3
```

Within an organizational account, an invited user (also known as a *member*) can have access to multiple workspaces within that organizational account.

```mermaid theme={null}
graph LR
    User1[User 1]
    User2_Org1[User 2]
    User2_Org2[User 2]
    User3_Org1[User 3] 
    User3_Org2[User 3]
    User3_Org3[User 3]

    Org1[Organization 1]
    Org2[Organization 2]
    Org3[Organization 3]

    WS1A[Workspace 1A]
    WS2A[Workspace 2A]
    WS2B[Workspace 2B]
    WS3A[Workspace 3A]
    WS3B[Workspace 3B]
    WS3C[Workspace 3C]
    
    Org1 --> WS1A
    Org1 -.-> User1
    Org1 -.-> User2_Org1
    Org1 -.-> User3_Org1
    User1 -.-> WS1A
    User2_Org1 -.-> WS1A
    User3_Org1 -.-> WS1A
    
    Org2 --> WS2A
    Org2 --> WS2B
    Org2 -.-> User2_Org2
    Org2 -.-> User3_Org2
    User2_Org2 -.-> WS2A
    User2_Org2 -.-> WS2B
    User3_Org2 -.-> WS2B
    
    Org3 --> WS3A
    Org3 --> WS3B
    Org3 --> WS3C
    Org3 -.-> User3_Org3
    User3_Org3 -.-> WS3A
    User3_Org3 -.-> WS3B
    User3_Org3 -.-> WS3C
```

Each user must be explicitly added to the organizational accounts and the workspaces within those accounts that they are allowed to access.

### Personal account architecture for Business SaaS

For Business SaaS accounts, each user also has a *personal account*
that is separate from any organizational accounts they can access.
Each personal account includes a *personal workspace* that operates
independently of organizational workspaces and other personal
workspaces. Each user can add a separate billing method that applies
only to usage within that personal account.

```mermaid theme={null}
graph LR
    User1[User 1]
    User2[User 2]
    User3[User 3]
    PA1[Personal Account for User 1]
    PA2[Personal Account for User 2]
    PA3[Personal Account for User 3]
    Org1[Organization 1]
    Org2[Organization 2]
    Org3[Organization 3]

    User1 --> PA1
    User2 --> PA2
    User3 --> PA3
    User1 -.-> Org1
    User2 -.-> Org1
    User2 -.-> Org2
    User3 -.-> Org1
    User3 -.-> Org2
    User3 -.-> Org3
```

## Create an organizational account

Only authorized Unstructured representatives can create organizational accounts. To request an organizational account to be created, contact your Unstructured
sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

## Access an organizational account

1. Sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that you want to access.

   If the organizational account selector is not shown, or if it does
   not list the organizational account that you want to access, your
   user has not been added to that organizational account.  In that case, the following message is displayed:

   *You're not part of an Unstructured organization
   yet. To use this platform, ask your administrator to add you to an
   organization.*

   To be added to the organizational account, contact one of its
   administrators.

## Add a member to an organizational account

<Info>
  Your user must have the **Super Administrator** [role](/pipelines/account/roles) in the organizational account to add members to that organizational account.
</Info>

To add a user to an organizational account as a member:

1. Sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that you want to add the member to.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon.
4. Click **Manage Account**.
5. In the top navigation bar, click **Members**.
6. Click **New Member**.
7. In the **Add New Member** dialog, enter the email address for the organizational account's new member.
8. In the role selector, select the new member's [role](/pipelines/account/roles) for this organizational account.
9. Click **Continue**.

## Change an organizational account role for a member

<Info>
  Your user must have the **Super Administrator** role in the organizational account to change its members' organizational account roles.
</Info>

1. Sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the member you want to change roles for.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon.
4. Click **Manage Account**.
5. In the top navigation bar, click **Members**.
6. Click the member you want to change roles for.
7. Next to **Role** click the **Edit** (pencil) icon.
8. Select the new role.
9. Click the **Accept** (check mark) icon.

## Remove a member from an organizational account

<Warning>
  Removing a member from an organizational account also removes the member from all of the organizational account's workspaces. The member will no longer
  have access to the organizational account or any of its workspaces. For Business SaaS accounts only, the member will continue to be able to use their personal account and personal workspace, which remain separate from the organizational account.

  Any connectors, workflows, jobs, or other resources that were created by the member within the organizational accounts' workspaces will not be removed.

  The only way to reverse removing a member from an organizational account is to add the member back to the organizational account—and to each of the workspaces that the member might have previously
  been a member of in that organizational account.
</Warning>

<Info>
  Your user must have the **Super Administrator** [role](/pipelines/account/roles) in the organizational account to remove members from that organizational account.
</Info>

1. Sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that you want to remove the member from.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon.
4. Click **Manage Account**.
5. In the top navigation bar, click **Members**.
6. In the list of members, click the member you want to remove.
7. In the member's settings pane, click **Remove Member**.
8. Confirm the removal by clicking **Continue**.

## Permanently delete an organizational account

<Warning>
  Deleting an organizational account is a permanent action and is not recoverable. Once an organizational account is deleted, all workspaces associated with that
  organizational account are also deleted and are not recoverable.

  The organizational account's members will no longer have access to the deleted organizational account or any of its workspaces.
  For Business SaaS accounts only, each member will continue to be able to use their personal account and personal workspace, which remain separate from the organizational account.

  Any code or scripts that reference connectors, workflows, jobs, or workflow API keys that are associated with workflows in the deleted
  organizational account will fail.
</Warning>

Only authorized Unstructured representatives can delete organizational accounts. To request an organizational account to be permanently deleted, contact your Unstructured
sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).


# Accounts overview
Source: https://docs.unstructured.io/pipelines/account/overview



After you sign in to your Unstructured account, you can view your account's [usage](/pipelines/account/usage) details.
If you are signing in to an Unstructured **Pay-As-You-Go** account, you can also manage your account's [billing](/pipelines/account/billing) details.

You can also create and manage your account's [API keys](/pipelines/account/api-key-url)—and get any [API URLs](/pipelines/account/api-key-url) as
needed—to work programmatically with your Unstructured account through the [Unstructured API](/api-reference/overview).

An Unstructured [organizational account](/pipelines/account/organizations) is a special kind of account that can manage multiple
[workspaces](/pipelines/account/workspaces) and use [roles](/pipelines/account/roles) to manage user access to the account and its workspaces.
Unstructured organizational accounts apply only to Unstructured **Business** accounts.


# Role-based access control (RBAC)
Source: https://docs.unstructured.io/pipelines/account/roles



<Note>
  The following information applies only to Unstructured [organizational accounts](/pipelines/account/organizations), which are
  available only for Unstructured **Business** accounts. To upgrade from an Unstructured **Let's Go** or **Pay-As-You-Go** account
  to a **Business** account, contact your Unstructured sales representative, or
  email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

*Roles* in Unstructured are part of the *role-based access control* (RBAC) system that manages permissions for members of
organizational accounts and their workspaces. For Business SaaS accounts, roles do not apply to personal accounts or personal workspaces.

Any member with the **Super Administrator** role in an organizational account can manage the roles of that organizational account's members and of members in the
organizational account's workspaces.

Any member with the **Super Administrator** role in an organizational account or the **Workspace Admin** role in a workspace within an
organizational account can manage the roles of that workspace's members.

A **Super Administrator** assigns an organizational account member's
initial role when the member is [added to the organizational account](/pipelines/account/organizations#add-a-member-to-an-organizational-account).
That role can be [changed](/pipelines/account/organizations#change-an-organizational-account-role-for-a-member) later.

A **Super Administrator** or **Workspace Admin** assigns a workspace
member's initial role when the member is [added to the workspace](/pipelines/account/workspaces#add-a-member-to-a-workspace).
That role can be [changed](/pipelines/account/workspaces#change-a-workspace-role-for-a-member) later.

## Site administrator role

<Note>
  The site administrator role is available only in dedicated instance and in-VPC deployments of [Unstructured Business](/business/overview).
</Note>

The site administrator (site admin) is a deployment-level role, independent of any organizational account. It controls access to **Site Settings**, where deployment-wide configuration lives.

A site admin can:

* [Configure identity providers](/business/idp/configuration)

An IdP group grants site administrator access. See [Site administrator access](/business/idp/group-mappings#site-administrator-access).

## Organizational account roles

Organizational account roles include:

* **Super Administrator**: Has access to all permissions, and has access to all resources created in an organization.
* **Account Member**: Able to be added to workspaces with a workspace role.
* **Billing Administrator**: Able to view billing information, usage, and account members.

These roles include the following permissions:

| Permission                                                      | Super Administrator | Account Member   | Billing Administrator |
| --------------------------------------------------------------- | ------------------- | ---------------- | --------------------- |
| Organizational account management                               |                     |                  |                       |
| Can add members to the account                                  | <span>Yes</span>    | <span>No</span>  | <span>No</span>       |
| Can remove members from the account                             | <span>Yes</span>    | <span>No</span>  | <span>No</span>       |
| Can view members of the account                                 | <span>Yes</span>    | <span>Yes</span> | <span>Yes</span>      |
| Can view an account member's details                            | <span>Yes</span>    | <span>Yes</span> | <span>Yes</span>      |
| Can change an account member's role                             | <span>Yes</span>    | <span>No</span>  | <span>No</span>       |
| Billing                                                         |                     |                  |                       |
| Can view usage and costs                                        | <span>Yes</span>    | <span>No</span>  | <span>Yes</span>      |
| Can edit payment information                                    | <span>Yes</span>    | <span>No</span>  | <span>Yes</span>      |
| Workspaces                                                      |                     |                  |                       |
| Can create workspaces                                           | <span>Yes</span>    | <span>No</span>  | <span>No</span>       |
| Can add members to a workspace                                  | <span>Yes</span>    | <span>No</span>  | <span>No</span>       |
| Can view members of a workspace                                 | <span>Yes</span>    | <span>Yes</span> | <span>No</span>       |
| Can be added to a workspace                                     | <span>Yes</span>    | <span>Yes</span> | <span>Yes</span>      |
| Can take actions in a workspace (based on their workspace role) | <span>Yes</span>    | <span>Yes</span> | <span>No</span>       |

## Workspace roles

Workspace roles include:

* **Viewer**: Ability to view all connectors and workflows that exist in the workspace in a read-only capacity.
* **Operator**: Ability to create, run, schedule, and delete any workflows that exist in the workspace.
  Can view connectors but cannot create or edit them.
* **Developer**: Ability to create and edit all connectors and workflows that exist in the workspace.
* **Workspace Administrator**: Ability to manage users on the workspace (invite, remove or change roles) as well as edit the workspace.

These roles include the following permissions:

| Resource          | Action                                   | Viewer           | Operator         | Developer        | Workspace Administrator |
| ----------------- | ---------------------------------------- | ---------------- | ---------------- | ---------------- | ----------------------- |
| Workflows         | Read                                     | <span>Yes</span> | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Create                                   | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Edit                                     | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Delete                                   | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Run                                      | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Schedule                                 | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Save                                     | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Duplicate                                | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Activate and deactivate                  | <span>No</span>  | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
| Connectors        | Read                                     | <span>Yes</span> | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Create                                   | <span>No</span>  | <span>No</span>  | <span>Yes</span> | <span>Yes</span>        |
|                   | Edit                                     | <span>No</span>  | <span>No</span>  | <span>Yes</span> | <span>Yes</span>        |
|                   | Delete                                   | <span>No</span>  | <span>No</span>  | <span>Yes</span> | <span>Yes</span>        |
| AI Providers      | Can see and use a configured AI provider | <span>Yes</span> | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Can configure AI provider secrets        | <span>No</span>  | <span>No</span>  | <span>No</span>  | <span>Yes</span>        |
|                   | Can delete AI provider secrets           | <span>No</span>  | <span>No</span>  | <span>No</span>  | <span>Yes</span>        |
| Workspace members | Add members to the workspace             | <span>No</span>  | <span>No</span>  | <span>No</span>  | <span>Yes</span>        |
|                   | View the workspace's members             | <span>Yes</span> | <span>Yes</span> | <span>Yes</span> | <span>Yes</span>        |
|                   | Remove members from the workspace        | <span>No</span>  | <span>No</span>  | <span>No</span>  | <span>Yes</span>        |
|                   | Change the workspace members' roles      | <span>No</span>  | <span>No</span>  | <span>No</span>  | <span>Yes</span>        |

Super Administrators in an organizational account have complete access to all of the organizational account's workspaces, regardless of whether they
are a member of those workspaces.


# Account usage
Source: https://docs.unstructured.io/pipelines/account/usage



## View usage details for an Unstructured account

To view usage details for an Unstructured **Let's Go** or **Pay-As-You-Go** account, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. On the sidebar, click your user icon, and then click **Account Settings**.
3. In the top navigation bar, click the **Usage History** tab. Your account's usage details are displayed.

To view usage details for an Unstructured **Business** account, do the following:

1. If you are not already signed in, sign in to Unstructured.
2. In the top navigation bar, in the list of available accounts, select the name of the [organizational account](/pipelines/account/organizations) that you want to view usage details for.
3. In the top navigation bar, click the **Usage History** tab. The account's usage details are displayed.

## View usage details for a workspace within an organizational account

<Note>
  The following information applies only to Unstructured **Business** accounts.

  Unstructured **Let's Go** and **Pay-As-You-Go** accounts do not support multiple workspaces. See the
  [previous section](#view-usage-details-for-an-unstructured-account) to view all usage details for
  the single personal workspace within a **Let's Go** or **Pay-As-You-Go** account.
</Note>

To view usage details for a specific [workspace](/pipelines/account/workspaces) within an [organizational account](/pipelines/account/organizations), do the following:

1. If you are not already signed in, sign in to your Unstructured account.

2. In the top navigation bar, in the list of available accounts, select the name of the organizational account that contains the workspace you want to view usage details for.

3. In the top navigation bar, in the list of available workspaces for the selected account, select the name of the workspace you want to view usage details for.

   <Note>
     If a list of available workspaces is not visible in the top navigation bar, click the gear (**Settings**) icon on the sidebar, click **View All Workspaces**, and then click the arrow next to the workspace you want to view usage details for.
   </Note>

4. In the upper navigation bar, click the **Usage** tab.

## How is usage calculated for billing purposes?

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

The **Billing** tab shows your account's current billing rate per page.

To complete billing-related tasks in your account, see [Account billing](/pipelines/account/billing).


# Workspaces
Source: https://docs.unstructured.io/pipelines/account/workspaces



<Note>
  The following information applies to workspaces in Unstructured [organizational accounts](/pipelines/account/organizations), which are
  available only for Unstructured **Business** accounts. To upgrade from an Unstructured **Let's Go** or **Pay-As-You-Go** account
  to a **Business** account, contact your Unstructured sales representative, or
  email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
</Note>

A *workspace* is a collection of resources such as connectors, workflows, jobs, and API keys. Each workspace operates independently of all other workspaces.

## Workspace architecture

A workspace can belong either to an [organizational account](/pipelines/account/organizations) or, for Business SaaS accounts, to a user's personal account. Only Business SaaS accounts include personal accounts and personal workspaces.

Every organizational account begins with one default workspace and can have multiple workspaces. For Business SaaS accounts, a user's personal account always includes exactly one personal workspace.

```mermaid theme={null}
graph LR
    Org1[Organization 1]
    Org2[Organization 2]
    Org3[Organization 3]
    WS1A[Workspace 1A]
    WS2A[Workspace 2A]
    WS2B[Workspace 2B]
    WS3A[Workspace 3A]
    WS3B[Workspace 3B]
    WS3C[Workspace 3C]

    Org1 --> WS1A
    
    Org2 --> WS2A
    Org2 --> WS2B

    Org3 --> WS3A
    Org3 --> WS3B
    Org3 --> WS3C
```

Users can have access to multiple workspaces within an organizational account.

```mermaid theme={null}
graph LR
    User1[User 1]
    User2_Org1[User 2]
    User2_Org2[User 2]
    User3_Org1[User 3] 
    User3_Org2[User 3]
    User3_Org3[User 3]

    Org1[Organization 1]
    Org2[Organization 2]
    Org3[Organization 3]

    WS1A[Workspace 1A]
    WS2A[Workspace 2A]
    WS2B[Workspace 2B]
    WS3A[Workspace 3A]
    WS3B[Workspace 3B]
    WS3C[Workspace 3C]
    
    Org1 --> WS1A
    Org1 -.-> User1
    Org1 -.-> User2_Org1
    Org1 -.-> User3_Org1
    User1 -.-> WS1A
    User2_Org1 -.-> WS1A
    User3_Org1 -.-> WS1A
    
    Org2 --> WS2A
    Org2 --> WS2B
    Org2 -.-> User2_Org2
    Org2 -.-> User3_Org2
    User2_Org2 -.-> WS2A
    User2_Org2 -.-> WS2B
    User3_Org2 -.-> WS2B
    
    Org3 --> WS3A
    Org3 --> WS3B
    Org3 --> WS3C
    Org3 -.-> User3_Org3
    User3_Org3 -.-> WS3A
    User3_Org3 -.-> WS3B
    User3_Org3 -.-> WS3C
```

Each user must be explicitly added to each workspace that they are
allowed to access. For Business SaaS accounts, a user always has
access to the personal workspace in their personal account. Access to
an organizational workspace also requires the user to be added to that
workspace's parent organizational account.

To make programmatic API calls to a workspace instead of using Pipelines, an API key must first exist for that workspace. The caller then uses that API key to make API calls to that workspace.
Each API key is valid only for its related workspace.

## Create a workspace

To create a workspace, you must first have an available Unstructured [organizational account](/pipelines/account/organizations).

Currently, only Unstructured representatives can create organizational accounts. To request an organizational account to be created,
contact your Unstructured
sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

<Info>
  Your user must have the **Super Administrator** [role](/pipelines/account/roles) in the intended workspace's parent organizational account to create workspaces in that organizational account.

  For Business SaaS accounts, each personal account includes exactly
  one personal workspace. You cannot create additional workspaces in
  personal accounts.
</Info>

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that you want to add a workspace to.
3. In the sidebar, above your user icon, click the **Settings** (gear) icon.
4. Click **View All Workspaces**.
5. Click **New Workspace**.
6. In the **Create New Workspace** dialog, enter a name for the new workspace.
7. Click **Continue**.

## Access a workspace

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace you want to access.
3. In the workspace selector, select the name of the workspace that you want to access.

   If the workspace selector does not list the workspace that you want
   to access, your user has not been added to that workspace. For
   organizational workspaces, contact one of the workspace's
   administrators or its parent organizational account's
   administrators to request access.

## Add a member to a workspace

<Info>
  Your user must have either the **Super Administrator** [role](/pipelines/account/roles) in the workspace's parent organizational account or the **Workspace Admin** role in the
  workspace to add members to that workspace.

  The user to be added must also have the **Developer** role in the workspace's parent organizational account.

  For Business SaaS accounts, each user always has access to the
  personal workspace in their personal account.
</Info>

To add a user to a workspace as a member:

1. Make sure the account user that you want to add has already been added as a member of the workspace's parent organizational account.
   [Learn how](/pipelines/account/organizations#add-a-member-to-an-organization).
2. If you are not already signed in, sign in to your Unstructured account.
3. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace you want to add a member to.
4. In the workspace selector, select the name of the workspace that you want to add a member to.
5. In the sidebar, above your user icon, click the **Settings** (gear) icon.
6. Click **Manage Workspace**.
7. In the top navigation bar, click **Members**.
8. Click **Add New Member**.
9. In the **Add New Member** dialog, select one of the organizational account's members to add to this workspace as a member.
10. In the role selector, select the new team member's [role](/pipelines/account/roles) for this workspace.
11. Click **Continue**.

## Change a workspace role for a member

<Info>
  Your user must have either the **Super Administrator** role in the workspace's parent organizational account or the **Workspace Admin** role in the
  workspace to change the roles for that workspace's members.
</Info>

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace for the member you want to change roles for.
3. In the workspace selector, select the name of the workspace that contains the member you want to change roles for.
4. In the sidebar, above your user icon, click the **Settings** (gear) icon.
5. Click **Manage Workspace**.
6. In the top navigation bar, click **Members**.
7. Next to the member you want to change roles for, click the ellipses.
8. Click **Edit Permissions**.
9. Select the new role.
10. Click **Continue**.

## Remove a member from a workspace

<Warning>
  A member who is removed from a workspace will no longer have access to the workspace. For Business SaaS accounts only, the member will continue to be able to use their personal account and personal workspace, which remain separate from the workspace.

  Any connectors, workflows, jobs, or other resources that were created by the member within the workspace will not be removed.

  The only way to reverse removing a member from a workspace is to add the member back to the workspace.
</Warning>

<Info>
  Your user must have either the **Super Administrator** [role](/pipelines/account/roles) in the workspace's parent organizational account or the **Workspace Admin** role in the
  workspace to remove members from that workspace.

  For Business SaaS accounts, a user cannot be removed from the
  personal workspace in their personal account.
</Info>

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace you want to remove a member from.
3. In the workspace selector, select the name of the workspace that you want to remove a member from.
4. In the sidebar, above your user icon, click the **Settings** (gear) icon.
5. Click **Manage Workspace**.
6. In the top navigation bar, click **Members**.
7. In the list of members, click the ellipses next to the member that you want to remove.
8. Click **Remove Member**.
9. Confirm the removal by clicking **Continue**.

## Create an API key for a workspace

<Info>
  Your user must have either the **Super Administrator** [role](/pipelines/account/roles) in the workspace's parent organizational account or the **Workspace Admin** or **Developer** role in the
  workspace to create API keys for that workspace.
</Info>

To make programmatic API calls to a workspace instead of using Pipelines, an API key must first exist for that workspace, and then the caller uses that API key to make API calls to that workspace.

An API key can be used only with the workspace for which it was created.

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace you want to create an API key for.
3. In the workspace selector, select the name of the workspace that you want to create an API key for.
4. In the sidebar, above your user icon, click the **Settings** (gear) icon.
5. Click **Manage Workspace**.
6. In the top navigation bar, click **API Keys**.
7. Click **Generate New Key**.
8. Enter some name for the key, and click **Continue**.
9. To get the key's value, in the list of available keys, click the **Copy** icon next to the newly created key. The key's value is
   copies to your system's clipboard.

## Delete an API key for a workspace

An API key is valid only for the workspace for which it was created.

<Warning>
  Deleting an existing API key is a permanent operation. Deleted API keys cannot be recovered.

  You cannot use a deleted API key to call the Unstructured APIs.
</Warning>

<Info>
  Your user must have either the **Super Administrator** [role](/pipelines/account/roles) in the workspace's parent organizational account or the **Workspace Admin** or **Developer** role in the
  workspace to delete API keys from that workspace.
</Info>

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the organizational account selector, select the name of the organizational account that contains the workspace you want to delete an API key for.
3. In the workspace selector, select the name of the workspace that you want to delete an API key for.
4. In the sidebar, above your user icon, click the **Settings** (gear) icon.
5. Click **Manage Workspace**.
6. In the top navigation bar, click **API Keys**.
7. In the list of available keys, click the ellipses next to the key you want to delete.
8. Click **Delete**.
9. Confirm the deletion by clicking **Continue**.

## Permanently delete a workspace

<Warning>
  Deleting a workspace is a permanent action and is not recoverable. Once a workspace is deleted, all connectors, workflows, jobs, and
  API keys that are associated with that workspace are also deleted and are not recoverable.

  The workspace's members are not deleted, but they will no longer have access to the deleted workspace's connectors, workflows, jobs, and API keys.

  Any code or scripts that reference the deleted workspace's connectors, workflows, jobs, or workspace API keys will fail.
</Warning>

<Info>
  For Business SaaS accounts, a user's personal workspace cannot be
  deleted from their personal account.
</Info>

Only authorized Unstructured representatives can delete workspaces. To request a workspace to be permanently deleted, contact your Unstructured
sales representative, or email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).


# Supported connectors
Source: https://docs.unstructured.io/pipelines/connectors



Unstructured supports connecting to the following source and destination types.

```mermaid theme={null}
  flowchart LR
    Sources-->Unstructured-->Destinations
```

## Sources

* [Azure Blob Storage](/pipelines/sources/azure-blob-storage)
* [Box](/pipelines/sources/box)
* [Confluence](/pipelines/sources/confluence)
* [Couchbase](/pipelines/sources/couchbase)
* [Databricks Volumes](/pipelines/sources/databricks-volumes)
* [Dropbox](/pipelines/sources/dropbox)
* [Elasticsearch](/pipelines/sources/elasticsearch)
* [Google Cloud Storage](/pipelines/sources/google-cloud)
* [Google Drive](/pipelines/sources/google-drive)
* [IBM FileNet](/pipelines/sources/filenet)
* [Jira](/pipelines/sources/jira)
* [Kafka](/pipelines/sources/kafka)
* [MongoDB](/pipelines/sources/mongodb)
* [OneDrive](/pipelines/sources/onedrive)
* [OpenSearch](/pipelines/sources/opensearch)
* [Outlook](/pipelines/sources/outlook)
* [PostgreSQL](/pipelines/sources/postgresql)
* [S3](/pipelines/sources/s3)
* [Salesforce](/pipelines/sources/salesforce)
* [SharePoint](/pipelines/sources/sharepoint)
* [Slack](/pipelines/sources/slack)
* [Snowflake](/pipelines/sources/snowflake)
* [Teradata](/pipelines/sources/teradata-sql)
* [VAST](/pipelines/sources/vast)
* [Zendesk](/pipelines/sources/zendesk)

If your source is not listed here, you might still be able to connect Unstructured to it through scripts or code by using the
[Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) or the
[Unstructured Ingest Python library](/open-source/ingestion/python-ingest).
[Learn more](/open-source/ingestion/source-connectors/overview).

## Destinations

* [Astra DB](/pipelines/destinations/astradb)
* [Azure AI Search](/pipelines/destinations/azure-ai-search)
* [Azure Blob Storage](/pipelines/destinations/azure-blob-storage)
* [Couchbase](/pipelines/destinations/couchbase)
* [Databricks Volumes](/pipelines/destinations/databricks-volumes)
* [Delta Tables in Amazon S3](/pipelines/destinations/delta-table)
* [Delta Tables in Databricks](/pipelines/destinations/databricks-delta-table)
* [Elasticsearch](/pipelines/destinations/elasticsearch)
* [Google Cloud Storage](/pipelines/destinations/google-cloud)
* [IBM FileNet](/pipelines/destinations/filenet)
* [IBM watsonx.data](/pipelines/destinations/ibm-watsonxdata)
* [Kafka](/pipelines/destinations/kafka)
* [Milvus](/pipelines/destinations/milvus)
* [MotherDuck](/pipelines/destinations/motherduck)
* [MongoDB](/pipelines/destinations/mongodb)
* [Neo4j](/pipelines/destinations/neo4j)
* [OneDrive](/pipelines/destinations/onedrive)
* [OpenSearch](/pipelines/destinations/opensearch)
* [Pinecone](/pipelines/destinations/pinecone)
* [PostgreSQL](/pipelines/destinations/postgresql)
* [Qdrant](/pipelines/destinations/qdrant)
* [Redis](/pipelines/destinations/redis)
* [S3](/pipelines/destinations/s3)
* [Snowflake](/pipelines/destinations/snowflake)
* [Teradata](/pipelines/destinations/teradata-sql)
* [Teradata Vector Store](pipelines/destinations/teradata-vector-v2)
* [VAST](pipelines/destinations/vast)
* [Weaviate](/pipelines/destinations/weaviate)

If your destination is not listed here, you might still be able to connect Unstructured to it through scripts or code by using the
[Unstructured Ingest CLI](/open-source/ingestion/overview#unstructured-ingest-cli) or the
[Unstructured Ingest Python library](/open-source/ingestion/python-ingest).
[Learn more](/open-source/ingestion/destination-connectors/overview).


# Astra DB
Source: https://docs.unstructured.io/pipelines/destinations/astradb



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Astra DB.

The requirements are as follows.

<iframe title="YouTube video player" />

* An IBM Cloud account or DataStax account.

  * For an IBM Cloud account, [sign up](https://cloud.ibm.com/registration) for an IBMid, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your IBMid.
  * For a DataStax account, [sign up](https://astra.datastax.com/signup) for a DataStax account, and then [sign in](https://accounts.datastax.com/session-service/v1/login) to DataStax with your DataStax account.

* An Astra DB database in the DataStax account. To create a database:

  a. After you sign in to DataStax, click **Create database**.<br />
  b. Click the **Serverless (vector)** tile, if it is not already selected.<br />
  c. For **Database name**, enter some unique name for the database.<br />
  d. Select a **Provider** and a **Region**, and then click **Create database**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-classic/databases/manage-create.html).

* An application token for the database. To create an application token:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Overview** tab, under **Database Details**, in the **Application Tokens** tile, click **Generate Token**.<br />
  c. Enter some **Token description** and select and **Expiration** time period, and then click **Generate token**.<br />
  d. Save the application token that is displayed to a secure location, and then click **Close**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/administration/manage-application-tokens.html).

* A keyspace in the database. To create a keyspace:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select **Create keyspace**.<br />
  c. Enter some **Keyspace name**, and then click **Add keyspace**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-keyspaces.html#keyspaces).

* A collection in the keyspace.

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * An existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. If Unstructured generates embeddings,
      the new collection's name will be `u<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `u<short-workflow-id`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * For the source connector only, an existing collection is required.
  * For the destination connector only, an existing collection is not required. At runtime, the collection behavior is as follows:

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your keyspace. The new collection's name will be `unstructuredautocreated`.

  To create a collection yourself:

  a. After you sign in to DataStax, in the list of databases, click the name of the target database.<br />
  b. On the **Data Explorer** tab, in the **Keyspace** list, select the name of the target keyspace.<br />
  c. In the **Collections** list, select **Create collection**.<br />
  d. Enter some **Collection name**.<br />
  e. Turn on **Vector-enabled collection**, if it is not already turned on.<br />
  f. Choose a mode for **Embedding generation method**. See [Astra DB generated embeddings](#astra-db-generated-embeddings).<br />
  g. If you chose **Bring my own**, enter the number of dimensions for the embedding model that you plan to use.<br />
  h. For **Similarity metric**, select **Cosine**.<br />
  i. Click **Create collection**.<br />

  [Learn more](https://docs.datastax.com/en/astra-db-serverless/databases/manage-collections.html#create-collection).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Astra DB**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Collection Name**: The name of the collection in the keyspace. If no value is provided, see the beginning of this article for the behavior at run time.
* **Keyspace** (*required*): The name of the keyspace in the collection.
* **Batch Size**: The maximum number of records per batch. The default is `20` if not otherwise specified.
* **Flatten Metadata**: Check this box to flatten the metadata into each record.
  Specifically, when flattened, the metadata key values are
  brought to the top level of the element, and the `metadata` key itself is removed. By default, the metadata is not flattened.
* **Binary Encode Vectors**: Uncheck this box to upload vectors as a list of numbers. By default, vectors are uploaded as binary. This is more efficient, but it makes it harder to view and work with vectors in the UI.
* **Enable Lexical Search**: Check this box to enable lexical and hybrid search. See [Lexical search](#lexical-search).
* **Astra DB Generated Embeddings**: Check this box to use an Astra vectorize integration for embeddings. See [Astra DB generated embeddings](#astra-db-generated-embeddings).
* **API Endpoint** (*required*): The database's associated API endpoint.
* **Token** (*required*): The application token for the database.

## Lexical search

When **Enable Lexical Search** is enabled, document text will be inserted into the `$lexical` field for lexical and hybrid search capabilities.

<Note>
  Your collection must be configured for lexical search, and your database must be in one of the supported AWS regions. For more information, see [Find data with lexicographical matching](https://docs.datastax.com/en/astra-db-serverless/databases/lexical-search.html) in the Astra DB documentation. Otherwise, you will encounter the following error when running a job:

  ` The Collection <keyspace>.<collection> does not have Lexical feature enabled. (LEXICAL_NOT_ENABLED_FOR_COLLECTION) [with 0 inserted ids])`
</Note>

## Astra DB generated embeddings

When **Astra DB Generated Embeddings** is enabled:

* Your document text will be inserted into the `$vectorize` field.
* Astra DB will automatically populate the `$vector` field with embeddings.
* Your workflow **must not** include an embedder node.
* You **must** have an embedding provider configured for your collection.

When **Astra DB Generated Embeddings** is disabled (the default):

* Unstructured will insert embeddings directly into the `$vector` field
* Your workflow **must** include an embedder node.

For more information, see [Auto-generate embeddings with vectorize](https://docs.datastax.com/en/astra-db-serverless/databases/embedding-generation.html) in the Astra DB documentation.

### Troubleshooting

You might encounter the following errors when running a job that uses this connector:

**Error**:

`Unable to vectorize data, embedding service not configured for the collection : <collection> (EMBEDDING_SERVICE_NOT_CONFIGURED) [with 0 inserted ids])`

You've enabled **Astra DB Generated Embeddings**, but your collection does not have an embedding provider. Either disable **Astra DB Generated Embeddings** in your connector settings, or recreate your collection with an embedding provider.

**Error**:

`Cannot use Unstructured embeddings and Astra-generated embeddings simultaneously. Please disable Astra generated embeddings or remove the Unstructured embedder.`

**or**:

`No vectors provided. Please enable an Unstructured embedding provider or configure Astra to generate embeddings.`

You must choose either Unstructured or Astra DB embeddings. Configuring neither, or both, is unsupported.


# Azure AI Search
Source: https://docs.unstructured.io/pipelines/destinations/azure-ai-search



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Azure AI Search.

The following video shows how to fulfill the minimum set of Azure AI Search requirements:

<iframe title="YouTube video player" />

Here are some more details about these requirements:

* The endpoint and API key for Azure AI Search. [Create an endpoint and API key](https://learn.microsoft.com/azure/search/search-create-service-portal).
* The name of the index in Azure AI Search. [Create an index](https://learn.microsoft.com/rest/api/searchservice/create-index).

  <iframe title="YouTube video player" />

  The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents
  that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  You can adapt the following index schema example for your own needs. Be sure to replace `<number-of-dimensions>`
  (in three locations in the following example) with the number of dimensions of the embedding model you are using:

  ```json theme={null}
  {
    "name": "elements-index",
    "fields": [
      {
        "name": "id",
        "type": "Edm.String",
        "key": true
      },
      {
        "name": "record_id",
        "type": "Edm.String",
        "filterable": true
      },
      {
        "name": "element_id",
        "type": "Edm.String"
      },
      {
        "name": "text",
        "type": "Edm.String",
        "searchable": true
      },
      {
        "name": "type",
        "type": "Edm.String"
      },
      {
        "name": "metadata",
        "type": "Edm.ComplexType",
        "fields": [
          {
            "name": "orig_elements",
            "type": "Edm.String"
          },
          {
            "name": "category_depth",
            "type": "Edm.Int32"
          },
          {
            "name": "parent_id",
            "type": "Edm.String"
          },
          {
            "name": "attached_to_filename",
            "type": "Edm.String"
          },
          {
            "name": "filetype",
            "type": "Edm.String"
          },
          {
            "name": "last_modified",
            "type": "Edm.DateTimeOffset"
          },
          {
            "name": "is_continuation",
            "type": "Edm.Boolean"
          },
          {
            "name": "file_directory",
            "type": "Edm.String"
          },
          {
            "name": "filename",
            "type": "Edm.String"
          },
          {
            "name": "data_source",
            "type": "Edm.ComplexType",
            "fields": [
              {
                "name": "url",
                "type": "Edm.String"
              },
              {
                "name": "version",
                "type": "Edm.String"
              },
              {
                "name": "date_created",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "date_modified",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "date_processed",
                "type": "Edm.DateTimeOffset"
              },
              {
                "name": "permissions_data",
                "type": "Edm.String"
              },
              {
                "name": "record_locator",
                "type": "Edm.String"
              }
            ]
          },
          {
            "name": "coordinates",
            "type": "Edm.ComplexType",
            "fields": [
              {
                "name": "system",
                "type": "Edm.String"
              },
              {
                "name": "layout_width",
                "type": "Edm.Double"
              },
              {
                "name": "layout_height",
                "type": "Edm.Double"
              },
              {
                "name": "points",
                "type": "Edm.String"
              }
            ]
          },
          {
            "name": "languages",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "page_number",
            "type": "Edm.String"
          },
          {
            "name": "links",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "page_name",
            "type": "Edm.String"
          },
          {
            "name": "link_urls",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "link_texts",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "sent_from",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "sent_to",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "subject",
            "type": "Edm.String"
          },
          {
            "name": "section",
            "type": "Edm.String"
          },
          {
            "name": "header_footer_type",
            "type": "Edm.String"
          },
          {
            "name": "emphasized_text_contents",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "emphasized_text_tags",
            "type": "Collection(Edm.String)"
          },
          {
            "name": "text_as_html",
            "type": "Edm.String"
          },
          {
            "name": "regex_metadata",
            "type": "Edm.String"
          },
          {
            "name": "detection_class_prob",
            "type": "Edm.Double"
          }
        ]
      },
      {
        "name": "embeddings",
        "type": "Collection(Edm.Single)",
        "dimensions": <number-of-dimensions>,
        "vectorSearchProfile": "embeddings-config-profile"
      }
    ],
    "vectorSearch": {
      "algorithms": [
        {
          "name": "hnsw-<number-of-dimensions>",
          "kind": "hnsw",
          "hnswParameters": {
            "m": 4,
            "efConstruction": 400,
            "efSearch": 500,
            "metric": "cosine"
          }
        }
      ],
      "profiles": [
        {
          "name": "embeddings-config-profile",
          "algorithm": "hnsw-<number-of-dimensions>"
        }
      ]
    },
    "semantic": {
      "configurations": [
        {
          "name": "default-semantic-config",
          "prioritizedFields": {
            "titleField": null,
            "prioritizedContentFields": [
              { "fieldName": "text" }
            ],
            "prioritizedKeywordsFields": []
          }
        }
      ]
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [Search indexes in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-an-index)
  * [Schema of a search index](https://learn.microsoft.com/azure/search/search-what-is-an-index#schema-of-a-search-index)
  * [Example index schema](https://learn.microsoft.com/rest/api/searchservice/create-index#examples)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

<h2>
  Set up Enterprise Connect authentication
</h2>

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see [Enterprise Connect for Azure](/business/azure/enterprise-connect).

To configure an Azure AI Search connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

1. Enable role-based access on your Azure AI Search service.

   <Warning>
     Azure AI Search defaults to API key authentication only. Complete this step to configure the service to accept the Microsoft Entra ID tokens that Enterprise Connect uses.
   </Warning>

   Follow the instructions in [Enable role-based access control for Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-security-enable-roles) in the Azure AI Search documentation. In the Azure portal, navigate to your search service, select **Settings** > **Keys**, and select **Both** to allow both API key and role-based authentication. If you want to use role-based authentication exclusively, select **Role-based access control** instead.

2. Create an App Registration for Unstructured in Microsoft Entra ID.

   In your Azure subscription, follow the instructions in [How to register an app in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, `unstructured-connector`). For **Supported account types**, select **Single tenant only**.

   You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the [single-tenant scenario](https://learn.microsoft.com/en-us/entra/identity-platform/single-and-multi-tenant-apps) as defined by Microsoft.

3. Add a federated identity credential to the App Registration.

   Follow the instructions in [Configure an app to trust an external identity provider](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust) in the Microsoft Entra documentation. Navigate to your App Registration, select **Certificates & secrets** in the left navigation pane, select the **Federated credentials** tab, and select **Add credential**.

   For **Federated credential scenario**, select **Other issuer**.

   Set the following values:

   | Field        | Value                                                                                                                                                                                                                                                                                                            |
   | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**     | A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.                                                                                                                                                                                     |
   | **Issuer**   | The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)                                                                                                                                                       |
   | **Subject**  | The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error. |
   | **Audience** | Set this to `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                                        |

   Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

4. Add a role assignment to grant your App Registration access to your Azure AI Search service.

   See [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) in the Azure documentation. Use the following values:

   * **Scope**: the Azure AI Search service that contains the index you want the connector to access.
   * **Role**: assign both **Search Index Data Contributor** (required for document indexing) and **Search Service Contributor** (required for connection testing). Repeat the role assignment steps for each role.
   * **Members**: select **User, group, or service principal**, then search for and select the App Registration you created in Step 2.

   When you reach the **Review + assign** tab, click **Review + assign** to complete the assignment.

5. Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the **Overview** page of your App Registration in the [Microsoft Entra admin center](https://entra.microsoft.com).

   * The **Tenant ID** (also called Directory ID) for your Azure subscription.
   * The **Client ID** of your App Registration.

Next, create the connector by following the steps in the **Create the destination connector** section on this page.

## Create the destination connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Azure AI Search**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Endpoint** (*required*): The endpoint URL for your Azure AI Search service, in the format `https://<service-name>.search.windows.net`.
* **Index Name** (*required*): The name of the index in your Azure AI Search service.
* **Authentication Method** (*required*): The authentication method to use. Select **API Key** or **Enterprise Connect (Workload Identity)**.

  **API Key only:**

  * **API Key** (*required*): The admin API key for your Azure AI Search service.

  **Enterprise Connect only:**

  * **Tenant ID** (*required*): The Tenant ID (also called Directory ID) for your Azure subscription.
  * **Client ID** (*required*): The Client ID of your App Registration.


# Azure Blob Storage
Source: https://docs.unstructured.io/pipelines/destinations/azure-blob-storage



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Azure Blob Storage.

The following video shows how to fulfill the minimum set of Azure Storage account requirements:

<iframe title="YouTube video player" />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  * **Read** and **List** for reading from the container only.
  * **Write** and **List** for writing to the container only.
  * **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these requirements:

* An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

  <iframe title="YouTube video player" />

* An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

  <iframe title="YouTube video player" />

  <iframe title="YouTube video player" />

* The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

  Create an SAS token (recommended):

  <iframe title="YouTube video player" />

  Get an access key or connection string:

  <iframe title="YouTube video player" />

<h2>
  Set up Enterprise Connect authentication
</h2>

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see [Enterprise Connect for Azure](/business/azure/enterprise-connect).

To configure an Azure Blob Storage connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

1. Create an App Registration for Unstructured in Microsoft Entra ID.

   In your Azure subscription, follow the instructions in [How to register an app in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, `unstructured-connector`). For **Supported account types**, select **Single tenant only**.

   You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the [single-tenant scenario](https://learn.microsoft.com/en-us/entra/identity-platform/single-and-multi-tenant-apps) as defined by Microsoft.

2. Add a federated identity credential to the App Registration.

   Follow the instructions in [Configure an app to trust an external identity provider](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust) in the Microsoft Entra documentation. Navigate to your App Registration, select **Certificates & secrets** in the left navigation pane, select the **Federated credentials** tab, and select **Add credential**.

   For **Federated credential scenario**, select **Other issuer**.

   Set the following values:

   | Field        | Value                                                                                                                                                                                                                                                                                                            |
   | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**     | A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.                                                                                                                                                                                     |
   | **Issuer**   | The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)                                                                                                                                                       |
   | **Subject**  | The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error. |
   | **Audience** | Set this to `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                                        |

   Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

3. Add a role assignment to grant your App Registration access to your Azure Blob Storage account.

   See [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) in the Azure documentation. Use the following values:

   * **Scope**: the Azure Blob Storage account that contains the data you want the connector to access.
   * **Role**: select **Storage Blob Data Reader** for a source, or **Storage Blob Data Contributor** for a destination.
   * **Members**: select **User, group, or service principal**, then search for and select the App Registration you created in Step 1.

   When you reach the **Review + assign** tab, click **Review + assign** to complete the assignment.

4. Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the **Overview** page of your App Registration in the [Microsoft Entra admin center](https://entra.microsoft.com).

   * The **Tenant ID** (also called Directory ID) for your Azure subscription.
   * The **Client ID** of your App Registration.

Next, create the connector by following the steps in the **Create the  connector** section on this page.

## Create the destination connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Click **New** or **Create Connector**.
4. For **Name**, enter a unique name for this connector.
5. In the **Provider** area, click **Azure Blob Storage**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Recursive** (source connector only): Check this box to recursively access files from subfolders within the container.

* **Remote URL** (*required*): The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* **Account Name**: For SAS token authentication, account key authentication, and Enterprise Connect (Workload Identity Federation), the Azure Storage account name.

* **SAS Token**: For SAS token authentication, the SAS token for the Azure Storage account.

* **Account Key**: For account key authentication, the key for the Azure Storage account.

* **Connection String**: For connection string authentication, the connection string for the Azure Storage account.

If you're using **Enterprise Connect (Workload Identity Federation)** ([dedicated instance](/business/dedicated-instances/overview) only), fill in the following fields:

* For **Authentication Method**, select **Enterprise Connect (Workload Identity Federation)**.
* For **Tenant ID**, enter the Tenant ID (also called Directory ID) for your Azure subscription.
* For **Client ID**, enter the Client ID of your App Registration.


# Couchbase
Source: https://docs.unstructured.io/pipelines/destinations/couchbase



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Couchbase.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Couchbase Capella clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe title="YouTube video player" />

For Couchbase Capella, you will need:

* A [Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier).
* A [Couchbase Capella cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started).
* A [bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket),
  [scope](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-scope),
  and [collection](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-collection)
  on the cluster.
* The cluster's [public connection string](https://docs.couchbase.com/cloud/get-started/connect.html#connect-from-sdk-cbsh-cli-or-ide).
* The [cluster access name (username) and secret (password)](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials).
* [Incoming IP address allowance](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) for the cluster.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

For a local Couchbase server, you will need:

* [Installation of a local Couchbase server](https://docs.couchbase.com/server/current/getting-started/start-here.html).
* [Connection details](https://docs.couchbase.com/server/current/guides/connect.html) to the local Couchbase server.

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Couchbase**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bucket** (*required*): The name of the bucket to connect to on the cluster.
* **Connection string** (*required*): The connection string for the Couchbase cluster.
* **Scope**: The name of the scope in the bucket. The default is `_default` if not otherwise specified.
* **Collection**: The name of the collection in the scope. The default is `_default` if not otherwise specified.
* **Batch Size**: The maximum number of documents per transmitted batch. The default is `50` if not otherwise specified.
* **Collection ID key** (source connector only): The name of the collection field that contains the document ID. The default is `id` if not otherwise specified
* **Username** (*required*): The cluster access name (username) that has access to the cluster.
* **Password** (*required*): The password for the corresponding username.

## Learn more

* <Icon icon="blog" />  [Couchbase Integration in Unstructured Platform](https://unstructured.io/blog/couchbase-integration-in-unstructured-platform)


# Delta Tables in Databricks
Source: https://docs.unstructured.io/pipelines/destinations/databricks-delta-table

Send processed data from Unstructured to a Delta Table in Databricks.

<Tip>
  This article covers connecting Unstructured to Delta Tables in Databricks.

  For information about connecting Unstructured to Delta Tables in Amazon S3 instead, see
  [Delta Tables in Amazon S3](/pipelines/destinations/delta-table).

  For information about connecting Unstructured to Databricks Volumes instead, see
  [Databricks Volumes](/pipelines/destinations/databricks-volumes).
</Tip>

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* [Account and resources](#accounts-and-resources)
* [Unity Catalog resources](#unity-catalog-resources)
* [Authentication and permissions](#authentication-and-permissions)

### Accounts and resources

* A Databricks account on [Amazon Web Services (AWS)](https://docs.databricks.com/getting-started/free-trial.html), [Microsoft Azure (Azure)](https://learn.microsoft.com/azure/databricks/getting-started/), or [Google Cloud Platform (GCP)](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html), [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* One of the following compute resources within the workspace:

  * A SQL warehouse for [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html), [Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create), or [GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html).

    The following video shows how to create a SQL warehouse if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the warehouse's owner to use it:

    <iframe title="YouTube video player" />

  * An all-purpose cluster for [AWS](https://docs.databricks.com/compute/use-compute.html), [Azure](https://learn.microsoft.com/azure/databricks/compute/use-compute), or [GCP](https://docs.gcp.databricks.com/compute/use-compute.html).

    The following video shows how to create an all-purpose cluster if you do not already have one available, get its **Server Hostname** and **HTTP Path** values, and set permissions for someone other than the cluster's owner to use it:

    <iframe title="YouTube video player" />

* The SQL warehouse's or cluster's **Server Hostname** and **HTTP Path** values for [AWS](https://docs.databricks.com/integrations/compute-details.html),
  [Azure](https://learn.microsoft.com/azure/databricks/integrations/compute-details), or
  [GCP](https://docs.gcp.databricks.com/integrations/compute-details.html).

* Unity Catalog enabled in the workspace for [AWS](https://docs.databricks.com/data-governance/unity-catalog/get-started.html),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/get-started), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/get-started.html).

### Unity Catalog resources

Within Unity Catalog:

* A catalog for [AWS](https://docs.databricks.com/catalogs/create-catalog.html), [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html).
* A schema for [AWS](https://docs.databricks.com/schemas/create-schema.html), [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) within that catalog.

  <Note>
    A *schema* was formerly known as a *database* in Unity Catalog.
  </Note>
* A table for [AWS](https://docs.databricks.com/tables/managed.html), [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or [GCP](https://docs.gcp.databricks.com/tables/managed.html) within that schema.

  How you create and specify the table depends on whether you want document metadata stored as nested JSON in a single column, or have Unstructured flatten the metadata and store each field as a separate columns. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.

  * If you choose to store document metadata as a JSON blob, you can specify a table, or have the connector attempt to create a table for you automatically at run time.

    ***To specify an existing table***

    In the connector settings as described later in this article, specify the name of an existing table.

    The following video shows how to create a catalog, schema, and a table in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

    <iframe title="YouTube video player" />

    If you want to use an existing table or create one yourself beforehand, this table must contain at minimum the following column names and their data types:

    ```text theme={null}
    CREATE TABLE IF NOT EXISTS <catalog_name>.<schema_name>.<table_name> (
        id STRING NOT NULL PRIMARY KEY,
        record_id STRING NOT NULL,
        element_id STRING NOT NULL,
        text STRING,
        embeddings ARRAY<FLOAT>,
        type STRING,
        metadata VARIANT
    );
    ```

    <Info>
      In Databricks, a table's *schema* is different than a *schema* in a catalog-schema object relationship in Unity Catalog.
    </Info>

    <Info>
      The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
    </Info>

    ***To have the connector create a table***

    You can have the connector attempt to create a table for you automatically at run time. In the connector settings as described later in this article, do one of the following:

    * Specify the name of the table that you want the connector to attempt to create within the specified catalog and schema.
    * Leave the table name blank. The connector will attempt to create a table within the specified catalog and schema.

      For [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview), the table is named `u<short-workflow-id>`.
      For the [Unstructured Ingest CLI and Ingest Python library](/open-source/ingestion/overview), the table is named `unstructuredautocreated`.

    The connector will attempt to create the table on behalf of the related Databricks workspace user or Databricks managed service principal that is referenced in the connector settings, as described later in these requirements.
    If successful, the table's owner is set as the related Databricks workspace user or Databricks managed service principal. The owner will have all Unity Catalog
    privileges on the table by default. No other Databricks workspace users or Databricks managed service principals will have any privileges on the table by default.

    <Warning>
      If the table's parent schema is not owned by the same Databricks workspace user or Databricks managed service principal that is
      referenced in the connector settings, then you should grant the new table's owner the `CREATE TABLE` privilege on that parent schema
      before the connector attempts to create the table. Otherwise, table creation could fail.
    </Warning>

    <Note>
      Using dashes (`-`) in the names of catalogs, schemas, and tables might cause isolated issues with the connector. It is
      recommended to use underscores (`_`) instead of dashes in the names of catalogs, schemas, and tables.
    </Note>

  * If you choose to flatten the document metadata, you must create the table before you configure your connector. For more information, see [Storing document metadata](#storing-document-metadata) later in this topic.

### Authentication and permissions

* For Databricks managed service principal authentication (using Databricks OAuth M2M) to the workspace:

  * A Databricks managed service principal.
    This service principal must have the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse.
  * The service principal's **UUID** (or **Client ID** or **Application ID**) value.
  * The OAuth **Secret** value for the service principal.

  To get this information, see Steps 1-3 of the instructions for [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
  [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

  <Note>
    For Azure Databricks, this connector only supports Databricks managed service principals for authentication.
    Microsoft Entra ID managed service principals are not supported.
  </Note>

  The following video shows how to create a Databricks managed service principal if you do not already have one available:

  <iframe title="YouTube video player" />

* For Databricks personal access token authentication to the workspace, the
  Databricks personal access token value for
  [AWS](https://docs.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat#azure-databricks-personal-access-tokens-for-workspace-users), or
  [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html#databricks-personal-access-tokens-for-workspace-users).
  This token must be for the workspace user who
  has the appropriate access permissions to the catalog, schema, table, volume, and cluster or SQL warehouse,

  The following video shows how to create a Databricks personal access token if you do not already have one available:

  <iframe title="YouTube video player" />

* The Databricks workspace user or Databricks managed service principal must have the following *minimum* set of permissions and privileges to write to an
  existing volume or table in Unity Catalog. If the owner of these is that Databricks workspace user or Databricks managed service principal, then
  they will have all necessary permissions and privileges by default. If the owner is someone else, then the following permissions and privileges must be
  explicitly granted to them before using the connector:

  * To use an all-purpose cluster for access, `Can Restart` permission on that cluster. Learn how to check and set cluster permissions for
    [AWS](https://docs.databricks.com/compute/clusters-manage.html#compute-permissions),
    [Azure](https://learn.microsoft.com/azure/databricks/compute/clusters-manage#cluster-level-permissions), or
    [GCP](https://docs.gcp.databricks.com/compute/clusters-manage.html#compute-permissions).

  * To use a SQL warehouse for access, `Can use` permission on that SQL warehouse. Learn how to check and set SQL warehouse permissions for
    [AWS](https://docs.databricks.com/compute/sql-warehouse/create.html#manage-a-sql-warehouse),
    [Azure](https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/create#manage), or
    [GCP](https://docs.gcp.databricks.com/compute/sql-warehouse/create.html#manage-a-sql-warehouse).

  * To access a Unity Catalog volume, the following privileges:

    * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
    * `USE SCHEMA` on the volume's parent schema in Unity Catalog.
    * `READ VOLUME` and `WRITE VOLUME` on the volume.

    Learn how to check and set Unity Catalog privileges for
    [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
    [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
    [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

  * To access a Unity Catalog table, the following privileges:

    * `USE CATALOG` on the table's parent catalog in Unity Catalog.
    * `USE SCHEMA` on the table's parent schema in Unity Catalog.
    * To create a new table, `CREATE TABLE` on the table's parent schema in Unity Catalog.
    * If the table already exists, `MODIFY` and `SELECT` on the table.

    Learn how to check and set Unity Catalog privileges for
    [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
    [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
    [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

* Within Unity Catalog, a volume
  for [AWS](https://docs.databricks.com/volumes/utility-commands.html),
  [Azure](https://learn.microsoft.com/azure/databricks/volumes/utility-commands),
  or [GCP](https://docs.gcp.databricks.com/volumes/utility-commands.html). The volume can be in the same
  schema as the table, or the volume and table can be in separate schemas. In either case, both of these
  schemas must share the same parent catalog.

  <Note>
    Using dashes (`-`) in the names of volumes might cause isolated issues with the connector. It is
    recommended to use underscores (`_`) instead of dashes in the names of volumes.
  </Note>

  The following video shows how to create a catalog, schema, and a volume in Unity Catalog if you do not already have them available, and set privileges for someone other than their owner to use them:

  <iframe title="YouTube video player" />

  ## Storing document metadata

  Unstructured offers the following options for storing [document metadata](/concepts/document-elements#metadata) in the destination table:

  * Store the metadata as a single nested JSON field.
  * *Flatten* the metadata by writing each metadata field as its own typed, queryable column.

  Each of these approaches has its advantages. In general, storing the metadata as a JSON blob works for most use cases, including when:

  * Performing dot.notation queries on the stored JSON is sufficient for your needs.
  * Document metadata schemas vary across file sources. When flattening document metadata, Unstructured drops fields that do not match existing columns in the schema.
  * You want the connector to automatically generate the destination table. This option is not supported when flattening document metadata.

  Flattening the metadata and storing it in multiple columns is most useful when:

  * You want to query individual metadata fields directly using standard SQL, without JSON parsing.
  * The business intelligence or analytics tools you are using require columnar data.

  Storing metadata as a JSON blob is the default.

  To store metadata as a JSON blob, when configuring the connector uncheck **Flatten Metadata** (in the Unstructured Pipelines), or set `flatten_metadata` to `false` (in the Unstructured API).  To flatten the metadata, check **Flatten Metadata**, or set `flatten_metadata` to `true`.

  ### How Unstructured flattens metadata

  If you choose to have Unstructured flatten document metadata, you must create the Delta Table to use as the destination, and specify it when creating the connector. In order to prevent possible data loss, Unstructured will not automatically create a new table.

  Considerations to keep in mind when creating the Delta Table:

  * The table must contain a column for each metadata field you want to store. Any metadata field that does not have a corresponding column in the table is silently dropped, although the event is written to the logs. For more information, see [Logging and monitoring](/business/security-compliance/overview#logging-and-monitoring).
  * Do not declare metadata columns as `NOT NULL`. Missing metadata values are written as `NULL`.
  * Unstructured passes values through as their JSON-native type: strings, numbers, boolean, and so on. For example, no special formatting is applied to timestamp values.
  * Metadata fields that are lists are not further flattened. Lists remain single columns.

  ### Metadata flattening example

  The following example demonstrates how Unstructured flattens metadata into separate columns. Consider the following metadata:

  ```json theme={null}
  {
    "metadata": {
      "category_depth": 1,
      "data_source": {
        "url": "s3://my-source-bucket/path/chart-and-table.pdf",
        "version": "864ae40b0719e976e98ba0a7b9fcba92",
        "record_locator": {
          "protocol": "s3",
          "remote_file_path": "s3://my-source-bucket/path/"
        }
      },
      "languages": ["eng"]
    }
  }
  ```

  When flattening metadata, Unstructured generates a field name comprised of the full path to that field within the metadata structure, from the outermost object to the field itself. For example, `protocol`, which is included in the `record_locator` object, which is in turn within `data_source`, becomes `data_source_record_locator_protocol`:

  ```json theme={null}
  {
    "category_depth": 1,
    "data_source_url": "s3://my-source-bucket/path/chart-and-table.pdf",
    "data_source_version": "864ae40b0719e976e98ba0a7b9fcba92",
    "data_source_record_locator_protocol": "s3",
    "data_source_record_locator_remote_file_path": "s3://my-source-bucket/path/",
    "languages": ["eng"]
  }
  ```

## Creating the connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Delta Tables in Databricks**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Server Hostname** (*required*): The target Databricks cluster's or SQL warehouse's **Server Hostname** value.

* **HTTP Path** (*required*): The cluster's or SQL warehouse's **HTTP Path** value.

* **Token** (*required* for PAT authentication): For Databricks personal access token (PAT) authentication, the target Databricks user's PAT value.

* **UUID** and **OAuth Secret** (*required* for OAuth authentication): For Databricks OAuth machine-to-machine (M2M) authentication, the Databricks managed service principal's **UUID** (or **Client ID** or **Application ID**) and OAuth **Secret** (client secret) values.

* **Catalog** (*required*): The name of the catalog in Unity Catalog for the target volume and table in the Databricks workspace.

* **Database**: The name of the schema (formerly known as a database) in Unity Catalog for the target table. The default is `default` if not otherwise specified.

  If the target table and volume are in the same schema (formerly known as a database), then **Database** and **Schema** will have the same names.

* **Table Name**: The name of the target table in Unity Catalog.

  * If a table name is specified, but a table with that name does not exist within the specified schema (formerly known as a database), the connector attempts to create a table with that name within that schema.
  * If no table name is specified, the connector attempts to create a table named `u<short-workflow-id>` within the specified schema (formerly known as a database).

  See the beginning of this article for additional technical requirements before having the connector attempt to create a table.

* **Schema**: The name of the schema (formerly known as a database) in Unity Catalog for the target volume. The default is `default` if not otherwise specified.

  If the target volume and table are in the same schema (formerly known as a database), then **Schema** and **Database** will have the same names.

* **Volume** (*required*): The name of the target volume in Unity Catalog.

* **Volume Path**: Any target folder path inside of the volume to use instead of the volume's root. If not otherwise specified, processing occurs at the volume's root.

<Note>
  Using dashes (`-`) in the names of catalogs, schemas (formerly known as databases), tables, and volumes might cause isolated issues with the connector. It is
  recommended to use underscores (`_`) instead of dashes in the names of catalogs, schemas, tables, and volumes.
</Note>

* **Flatten Metadata**: Check to have Unstructured flatten the metadata and store each field as a separate columns. Uncheck to store document metadata as nested JSON in a single column. For more information, see [Storing document metadata](#storing-document-metadata).

## Learn more

* <Icon icon="blog" />  [Getting Started with Unstructured and Delta Tables in Databricks](https://unstructured.io/blog/getting-started-with-unstructured-and-delta-tables-in-databricks)
* <Icon icon="blog" />  [Integration Highlight: Databricks Delta Tables](https://unstructured.io/blog/integration-highlight-databricks-delta-tables)
* <Icon icon="blog" />  [RAG: Seamlessly Integrating Context from Multiple Sources into Delta Tables in Databricks](https://unstructured.io/blog/rag-seamlessly-integrating-context-from-multiple-sources-into-delta-tables-in-databricks)


# Databricks Volumes
Source: https://docs.unstructured.io/pipelines/destinations/databricks-volumes



<Tip>
  This article covers connecting Unstructured to Databricks Volumes.

  For information about connecting Unstructured to Delta Tables in Databricks instead, see
  [Delta Tables in Databricks](/pipelines/destinations/databricks-delta-table).
</Tip>

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Databricks Volumes.

The requirements are as follows.

* A Databricks account on [AWS](https://docs.databricks.com/getting-started/free-trial.html),
  [Azure](https://learn.microsoft.com/azure/databricks/getting-started/), or
  [GCP](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or
  [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* The workspace's URL. Get the workspace URL for
  [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
  [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
  or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

  Examples:

  * AWS: `https://<workspace-id>.cloud.databricks.com`
  * Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
  * GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

  <Note>
    Do not add a trailing slash (`/`) to the workspace URL.
  </Note>

* The Databricks authentication details. For more information, see the documentation for
  [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
  or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

  For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), the following Databricks authentication types are supported:

  * Databricks OAuth machine-to-machine (M2M) authentication for\
    [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

    You will need the **Client ID** (or **UUID** or **Application** ID) and OAuth **Secret** (client secret) values for the corresponding service principal.
    Note that for Azure, only Databricks managed service principals are supported. Microsoft Entra ID managed service principals are not supported.

    The following video shows how to create a Databricks managed service principal:

    <iframe title="YouTube video player" />

  * Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html).

    You will need the personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  For [Unstructured Ingest](/open-source/ingestion/overview), the following Databricks authentication types are supported:

  * For Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html): The personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  * For username and password (basic) authentication ([AWS](https://docs.databricks.com/archive/dev-tools/basic.html) only): The user's name and password values.

  * For OAuth machine-to-machine (M2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html)): The client ID and OAuth secret values for the corresponding service principal.

  * For OAuth user-to-machine (U2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-u2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-u2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-u2m.html)): No additional values.

  * For Azure managed identities (formerly Managed Service Identities (MSI) authentication) ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-mi) only): The client ID value for the corresponding managed identity.

  * For Microsoft Entra ID service principal authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-sp) only): The tenant ID, client ID, and client secret values for the corresponding service principal.

  * For Azure CLI authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-cli) only): No additional values.

  * For Microsoft Entra ID user authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/user-aad-token) only): The Entra ID token for the corresponding Entra ID user.

  * For Google Cloud Platform credentials authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-creds.html) only): The local path to the corresponding Google Cloud service account's credentials file.

  * For Google Cloud Platform ID authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-id.html) only): The Google Cloud service account's email address.

* The name of the parent catalog in Unity Catalog for
  [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
  [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
  [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html) for the volume.

* The name of the parent schema (formerly known as a database) in Unity Catalog for
  [AWS](https://docs.databricks.com/schemas/create-schema.html),
  [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
  [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) for the volume.

* The name of the volume in Unity Catalog for [AWS](https://docs.databricks.com/tables/managed.html),
  [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
  [GCP](https://docs.gcp.databricks.com/tables/managed.html), and optionally any path in that volume that you want to access directly, beginning with the volume's root.

* The Databricks workspace user or service principal must have the following *minimum* set of privileges to read from or write to the
  existing volume in Unity Catalog:

  * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
  * `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
  * `READ VOLUME` and `WRITE VOLUME` on the volume.

  The following videos show how to create and set privileges for a catalog, schema (formerly known as a database), and volume in Unity Catalog.

  <iframe title="YouTube video player" />

  Learn more about how to check and set Unity Catalog privileges for
  [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Databricks Volumes**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Host** (*required*): The Databricks workspace host URL.

  <Note>
    Do not add a trailing slash (`/`) to the host URL.
  </Note>

* **Catalog** (*required*): The name of the catalog to use.

* **Schema** : The name of the associated schema. If not specified, **default** is used.

* **Volume** (*required*): The name of the associated volume.

* **Volume Path** : Any optional path to access within the volume.

* For **Authentication Method**, if you select **Service Principal**, you must also specify the following:

  * **OAuth Secret** (*required*): The associated OAuth **Secret** value for the Databricks managed service principal that has the appropriate privileges to the volume.
  * **UUID** (*required*): The **Client ID** (or **UUID** or **Application ID**) value for the Databricks managed service principal that has appropriate privileges to the volume.

* For **Authentication Method**, if you select **Token**, you must also specify the Databricks personal access token's value in the **Token** field.


# Delta Tables in Amazon S3
Source: https://docs.unstructured.io/pipelines/destinations/delta-table



<Tip>
  This article covers connecting Unstructured to Delta Tables in Amazon S3. For information about
  connecting Unstructured to Delta Tables in Databricks instead, see
  [Delta Tables in Databricks](/pipelines/destinations/databricks-delta-table).
</Tip>

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to a Delta Table, stored in Amazon S3.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Amazon S3 requirements to store Delta Tables:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account.

For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* For authenticated bucket write access or both, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. To grant permissions,
  attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
  To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Provide an AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, provide the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Delta table output format

A Delta table consists of Parquet files that contain data and a transaction log that stores metadata about the transactions.
[Learn more](https://delta-io.github.io/delta-rs/how-delta-lake-works/architecture-of-delta-table/).

The Delta Tables in Amazon S3 destination connector generates the following output within the specified path to the S3 bucket (or the specified folder within the bucket):

* Initially, one Parquet (`.parquet`) file per file in the source location. For example, for a file in the source location named `my-file.pdf`, an associated
  file with the extension `.parquet` is generated. Various kinds of file transactions can result in additional Parquet files being generated. These Parquet filenames are automatically generated by the Delta Lake engine and are not meant to be manually modified.
* A folder named `_delta_log` that contains metadata and change history about the `.parquet` files. As Parquet files are added to, changed, or removed from
  the specified bucket or folder path, the `_delta_log` folder is updated with any related metadata and change history details.

Together, this set of Parquet files and their associated `_delta_log` folder (and its contents) describe a single, versioned Delta table. Because of this, Unstructured recommends the following usage best practices:

* In the source location, each set of source files that is to be considered as a unit for change management purposes should be controlled by a unique, dedicated
  Delta Tables in S3 destination connector. This connector should reference a unique, dedicated output folder within the bucket. Having
  multiple workflows refer to different sets of source files, yet all share the same Delta table, could results in data loss or table corruption.
* Avoid directly modifying, adding, or deleting Parquet data files or the `_delta_log` folder within a Delta table's directory. This can lead to data loss or table corruption.
* If you need to copy or move a Delta table to a different location,
  you must move or copy its entire set of Parquet files and its associated `_delta_log` folder (and its contents) together as a unit.
  Note that the copied or moved Delta table will
  no longer be controlled by the original Delta Tables in S3 destination connector.

## Create the destination connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Delta Table**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **AWS Region** (*required*): The AWS Region identifier (for example, `us-east-1`) for the Amazon S3 bucket you want to store the Delta Table in.
* **Bucket URI** (*required*): The URI of the Amazon S3 bucket you want to store the Delta Table in. This typically takes the format `s3://my-bucket/my-folder`.
* **AWS Access Key ID** (*required*): The AWS access key ID for the AWS IAM principal (such as an IAM user) that has the appropriate access to the S3 bucket.
* **AWS Secret Access Key** (*required*): The AWS secret access key for the corresponding AWS access key ID.


# Elasticsearch
Source: https://docs.unstructured.io/pipelines/destinations/elasticsearch



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Elasticsearch.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Elastic Cloud instances are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

* For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

  <iframe title="YouTube video player" />

* For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

  <iframe title="YouTube video player" />

* You will need the name of the index on the instance.

  For the destination connector, if you need to create an index, you can use for example the following `curl` command. Replace the following placeholders:

  * Replace `<host>:<port>` with the instance's host identifier and port number.

  * Replace one of the following, and remove the other:

    * For an instance's user and password combination, replace `<username>` with your Elasticsearch or Elastic Cloud instance's user name, replace `<password>` with your password, and then remove `--user "Authorization: ApiKey <your-api-key>"`.
    * For an Elastic Cloud API key, replace `<your-api-key>` with your Elastic Cloud API key, and then remove `--user "<username>:<password>"`.

  * Replace `<index-name>` with the name of the new index on the instance.

  * Replace `<index-schema>` with the schema for the index. A schema is optional; see the explanation
    following this `curl` command for more information.

  ```bash theme={null}
  curl --request PUT "<host>:<port>/<index-name>" \
  --user "<username>:<password>" \
  --user "Authorization: ApiKey <your-api-key>" \
  [--header "Content-Type: application/json" \
  --data '<index-schema>']
  ```

  To learn more, see [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

  For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
  Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
  to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  For objects in the `metadata` field that Unstructured produces and that you want to store in an Elasticsearch or Elastic Cloud index, you must create fields in your index's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    },
    "entities": {
      "items": [
        {
          "entity": "HOME FOR FUTURE",
          "type": "ORGANIZATION"
        },
        {
          "entity": "221 Queen Street, Melbourne VIC 3000",
          "type": "LOCATION"
        }
      ],
      "relationships": [
        {
          "from": "HOME FOR FUTURE",
          "relationship": "based_in",
          "to": "221 Queen Street, Melbourne VIC 3000"
        }
      ]
    }
  }
  ```

  You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
  required by Unstructured whenever you create your own index schema:

  * `element_id`
  * `record_id`, which is required by Unstructured for intelligent record updates.
  * `type`, which is not required, but highly recommended.
  * `text`
  * `embeddings` if embeddings are generated; make sure to set `dims` to the same number of dimensions as the embedding model generates.

  ```json theme={null}
  {
    "mappings": {
      "properties": {
        "element_id": {
          "type": "keyword"
        },
        "record_id": {
          "type": "text"
        },
        "text": {
          "type": "text"
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "embeddings": {
          "type": "dense_vector",
          "dims": 1536,
          "index": true,
          "similarity": "cosine"
        },
        "metadata": {
          "properties": {
            "is_extracted": {
              "type": "boolean"
            },
            "coordinates-points": {
              "type": "float"
            },
            "coordinates-system": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates-layout_width": {
              "type": "long"
            },
            "coordinates-layout_height": {
              "type": "long"
            },
            "filetype": {
              "type": "keyword"
            },
            "languages": {
              "type": "keyword"
            },
            "page_number": {
              "type": "integer"
            },
            "image_mime_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "data_source-url": {
              "type": "keyword"
            },
            "data_source-record_locator-protocol": {
              "type": "keyword"
            },
            "data_source-record_locator-remote_file_path": {
              "type": "keyword"
            },
            "entities-items": {
              "properties": {
                "entity": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "entities-relationships": {
              "properties": {
                "from": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "relationship": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "to": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
  * [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
  * [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  <iframe title="YouTube video player" />

* For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console or [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

  <Note>
    When generating an Elastic Cloud service instance's API key through the Elasticsearch Service web console, be sure after you click **Create API key** to select **Encoded** to get the API key's value.
    (The **Encoded** API key value is the Base64 encoding of the UTF-8 representation of the API key's ID and key value joined by a colon \[:].)
  </Note>

* For self-managed Elasticsearch, you will need:

  * The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).

  * If you're using basic authentication to the self-managed instance, the user's name and password.

  * If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

    <Note>
      When generating an API key for a self-managed Elasticsearch instance by calling [POST /\_security/api\_key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key), be sure to use
      the `encoded` value in the response payload. (This `encoded` value is the Base64 encoding of the UTF-8 representation of the payload's `id` and `api_key` values joined by a colon \[:]). Do not use the `api_key` value by itself.
    </Note>

  * If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Elasticsearch**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host** (*required*): The endpoint URL for the target cluster.
* **Index Name** (*required*): The name of the target index in the cluster.
* **API Key** (*required*): The Elastic Cloud API key for the target cluster.

## Learn more

* <Icon icon="blog" />  [Finding Needles in a Haystack: PII Detection at Scale with Unstructured, Box, and Elasticsearch](https://unstructured.io/blog/finding-needles-in-a-haystack-pii-detection-at-scale-with-unstructured-box-and-elasticsearch)


# IBM FileNet
Source: https://docs.unstructured.io/pipelines/destinations/filenet

Connect Unstructured to IBM FileNet as a destination to send processed document data into your FileNet content repository.

IBM FileNet is an enterprise content management (ECM) system that enables organizations to store, manage, secure, and automate documents and business content at scale.

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* An [IBM Cloud Pak for Business Automation as a Service](https://www.ibm.com/products/cloud-pak-for-business-automation) account with access to the IBM FileNet [Content Platform Engine server](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=architecture-content-platform-engine) object store to which you want to connect. IBM FileNet is a component of IBM Cloud Pak for Business Automation as a Service.

To access the information you'll need to configure the connector:

1. Log into your [account](https://www.automationcloud.ibm.com/auth/index.jsp).

2. Choose the **Navigator** tile.

   The [IBM Navigator](https://www.ibm.com/docs/en/content-navigator/3.2.0) launches in a separate browser window, and provides a view of your object stores and content. You can use the IBM Navigator views to find the information necessary to create a connection to Unstructured.

   For the URL of your IBM FileNet server:

   * The server URL displays in the browser address bar. You only need the base URL that specifies the company and domain. For example, `https://<company-name>.automationcloud.ibm.com`.

   For [object store](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=infrastructure-defining-object-stores) names and folder paths:

   * Select the folder in the left pane. The full folder path is displayed at the top of the main detail pane, in the following format: `<object-store>/<folder>/etc`.

   For the document class:

   * Right-click the document and select **Properties**.

   For the account username:

   * Right-click the profile icon on the upper right in the top menu.

## Creating the connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Click **New** or **Create Connector**.
4. For **Name**, enter a unique name for this connector.
5. In the **Provider** area, click **FileNet**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Server URL** (*required*): The base URL of your Content Platform Engine, containing both the IBM domain and your company's subdomain. For example,  `https://<company-name>.automationcloud.ibm.com`.
* **Object Store** (*required*): The name of the object store to connect within the server.
* **Folder Path** (*required*, source connector only): The path to the folder within the object store to use as the source.
* **Target Folder** (*required*, destination connector only): The path to the folder within the object store to use as the upload destination.
* **Document Class**: The class of documents to include.
* **Recursive**: (Source connector only) Select to include documents contained in any subfolders.
* **Username** (*required*): The username of the IBM Cloud Pak for Business Automation as a Service account to use.
* **Password** (*required*): The password for the corresponding username.


# Google Cloud Storage
Source: https://docs.unstructured.io/pipelines/destinations/google-cloud



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Google Cloud Storage.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

* A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
  [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:
    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```
  * For Windows:
    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

  This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

  * `Storage Object Viewer` for bucket read access.
  * `Storage Object User` for bucket write access.
  * The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

  To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
  in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Google GCS**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bucket URI** (*required*): The URI for the Google Cloud Storage bucket and any target folder path within the bucket. This URI takes the format `gs://<bucket-name>[/folder-name]`.
* **Recursive** (source connector only): Check this box to ingest data recursively from any subfolders, starting from the path specified by **Bucket URI**.
* **Service Account Key** (*required*): The contents of a service account key file, expressed as a single string without line breaks, for a Google Cloud service account that has the required access permissions to the bucket.


# IBM watsonx.data
Source: https://docs.unstructured.io/pipelines/destinations/ibm-watsonxdata



<Tip>
  The IBM watsonx.data destination connector relies on an Apache Iceberg-based catalog within the watsonx.data data store instance.
  Apache Iceberg is suitable for managed data storage and cataloging, but not for embedding storage or semantic similarity
  queries. For embedding storage and semantic similarity queries, Unstructured recommends that you use the following destination connectors
  instead:

  * [Astra DB](/pipelines/destinations/astradb)
  * [Milvus](/pipelines/destinations/milvus) on IBM watsonx.data
</Tip>

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to IBM watsonx.data.

The requirements are as follows.

* An [IBM Cloud account](https://cloud.ibm.com/login). [Create an IBM Cloud account](https://cloud.ibm.com/registration) if you do not already have one.

* An API key for the IBM Cloud account. If you do not have one already, create one as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. In the top navigation bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
  3. On the sidebar, under **Manage identities**, click **API keys**.
  4. With the **View** list showing **My IBM Cloud API keys**, click **Create**.
  5. Enter some **Name** and an optional **Description** for the API key.
  6. Leave **Leaked action** set to **Disable the leaked key** and **Session management** set to **No**.
  7. Click **Create**.
  8. Click **Copy** or **Download** to copy or save the API key to a secure location. You won't be able to access this API key from this screen again.

* An IBM Cloud Object Storage (COS) instance in the account, and a bucket within that instance. If you do not have them already,
  create them as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. Click **Create resource**.
  4. With **IBM Cloud catalog** selected, search for and select **Object Storage**.
  5. Complete the on-screen instructions to finish creating the COS instance.
  6. With the COS instance's settings page shown, on the **Buckets** tab, click **Create bucket**.
  7. Complete the on-screen instructions to finish creating the bucket.

* The name, region, and public endpoint for the target bucket within the target Cloud Object Storage (COS) instance. To get these:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Storage**, and then click the target COS instance.
  4. On the **Buckets** tab, click the target bucket.
  5. On the **Configuration** tab, note the following:

     * Under **Bucket details**, note the **Bucket name**. This is the bucket's name.
     * Under **Bucket details** section, note the value inside of the parentheses inside **Location**, for example `us-east`. This is the bucket's region.
     * Under **Endpoints**, note the value of **Public**, for example `s3.us-east.cloud-object-storage.appdomain.cloud`. (Ignore the values of
       **Private** and **Direct**). This is the bucket's public endpoint.

* An HMAC access key ID and secret access key for the target Cloud Object Storage (COS) instance. If you do not have them already,
  get or create them as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.

  3. In the list of resources, expand **Storage**, and then click the target COS instance.

  4. On the **Service credentials** tab, if there is a credential that you want to use in the list, expand the credential, and copy the following values to a secure location:

     * `access_key_id` under `cos_hmac_keys`, which represents the HMAC access key ID.
     * `secret_access_key` under `cos_hmac_keys`, which represents the HMAC secret access key.

     After you have copied the preceding values, you have completed this procedure.

  5. If there is not a credential that you want to use, or there are no credentials at all, click **New Credential**.

  6. Enter some **Name** for the credential.

  7. For **Role**, select at least **Writer**, leave **Select Service ID** set to **Auto Generated**,
     switch on **Include HMAC Credential**, and then click **Add**.

  8. In the list of credentials, expand the credential, and copy the following values to a secure location:

     * `access_key_id` under `cos_hmac_keys`, which represents the HMAC access key ID.
     * `secret_access_key` under `cos_hmac_keys`, which represents the HMAC secret access key.

* An IBM watsonx.data data store instance in the IBM Cloud account. If you do not have one already, create one as follows:

  <iframe title="YouTube video player" />

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. Click **Create resource**.
  4. With **IBM Cloud catalog** selected, search for and select **watsonx.data**.
  5. Complete the on-screen instructions to finish creating the watsonx.data data store instance.

* An Apache Iceberg-based catalog within the watsonx.data data store instance. If you do not have one already, create one as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).

  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.

  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.

  4. Click **Open web console**.

  5. If prompted, log in to the web console.

  6. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.

  7. Click **Add component**.

  8. Under **Storage**, click **IBM Cloud Object Storage**, and then click **Next**.

  9. Complete the on-screen instructions to finish creating the Iceberg catalog. This includes providing the following settings:

     If you select **Discover COS instance**, you must provide the following settings:

     * The name of the target COS instance.
     * The name of the target storage object (such as the target bucket) within the COS instance.
     * Some display name for the component.
     * After you provide this information, do the following:

       a. Check the box labelled **Associate Catalog**.<br />
       b. For **Catalog type**, select **Apache Iceberg**.<br />
       c. Enter some **Catalog name**.<br />
       d. Click **Associate**.<br />

     If you select **Register my own**, you must provide the following settings:

     * Some display name for the component.
     * The name of the target bucket within the target Cloud Object Storage (COS) instance that you noted earlier.
     * The region for the target bucket, which you noted earlier.
     * The public endpoint for the target bucket, which you noted earlier. For this screen only, be sure to prefix the public endpoint with `https://`.
     * The HMAC access key ID for the target COS instance, which you noted earlier.
     * The HMAC secret access key for the target COS instance, which you noted earlier.
     * After you provide this information, do the following:

       a. Check the box labelled **Associate Catalog**.<br />
       b. For **Catalog type**, select **Apache Iceberg**.<br />
       c. Enter some **Catalog name**.<br />
       d. Click **Associate**.<br />

  10. On the sidebar, click **Infrastructure manager**. Make sure the catalog is associated with the appropriate engines. If it is not, rest your mouse
      on an unassociated target engine, click the **Manage associations** icon, check the box next to the target catalog's name, and then
      click **Save and restart engine**.

      To create an engine if one is not already shown, click **Add component**, and follow the on-screen to add an appropriate engine from the list of available **Engines**
      (for example, an **IBM Presto** engine).

* The catalog name and metastore REST endpoint for the target Iceberg catalog. To get this:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. In the **Catalogs** section, click the target Iceberg catalog.
  8. On the **Details** tab, note the value of **Name** representing the catalog name, and **Metastore REST endpoint** representing the metastore REST endpoint. (Ignore the **Metastore Thrift endpoint** value.)

* A namespace (also known as a schema) and a table in the target catalog. If you do not have these already, create them as follows:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Data manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. On the **Browse data** tab, under **Catalogs associated**, click the target catalog.
  8. Click the ellipses, and then click **Create schema**.
  9. Enter some **Name** for the schema, and then click **Create**.
  10. On the sidebar, click **Query workspace**.
  11. In the SQL editor, enter and run a table creation statement such as the following one that uses
      [Presto SQL](https://prestodb.io/docs/current/connector/iceberg.html) syntax, replacing `<catalog-name>` with the name of the target
      catalog and `<schema-name>` with the name of the target schema, with at minimum the fields `id`, `type`, `text`, `element_id`, and `record_id`:

      ```sql theme={null}
      CREATE TABLE <catalog-name>.<schema-name>.elements (
         id varchar,
         type varchar,
         text varchar,
         element_id varchar,
         record_id varchar,
         <additional-fields-as-needed>
      )
      WITH (
         delete_mode = 'copy-on-write',
         format = 'PARQUET',
         format_version = '2'
      )
      ```

      <Info>
        The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
      </Info>

      For objects in the `metadata` field that Unstructured produces and that you want to store in an Apache Iceberg table in IBM watsonx.data,
      you must create fields in your table schema that follows Unstructured's `metadata` field naming convention. For example, if Unstructured
      produces a `metadata` field with the following child objects:

      ```json theme={null}
      "metadata": {
        "is_extracted": "true",
        "coordinates": {
          "points": [
            [
              134.20055555555555,
              241.36027777777795
            ],
            [
              134.20055555555555,
              420.0269444444447
            ],
            [
              529.7005555555555,
              420.0269444444447
            ],
            [
              529.7005555555555,
              241.36027777777795
            ]
          ],
          "system": "PixelSpace",
          "layout_width": 1654,
          "layout_height": 2339
        },
        "filetype": "application/pdf",
        "languages": [
          "eng"
        ],
        "page_number": 1,
        "image_base64": "aaa...bbb...111",
        "image_mime_type": "image/jpeg",
        "filename": "realestate.pdf",
        "data_source": {
          "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
          "record_locator": {
            "protocol": "file",
            "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
          }
        },
        "entities": {
          "items": [
            {
              "entity": "HOME FOR FUTURE",
              "type": "ORGANIZATION"
            },
            {
              "entity": "221 Queen Street, Melbourne VIC 3000",
              "type": "LOCATION"
            }
          ],
          "relationships": [
            {
              "from": "HOME FOR FUTURE",
              "relationship": "based_in",
              "to": "221 Queen Street, Melbourne VIC 3000"
            }
          ]
        }
      }
      ```

      You could create corresponding fields in your table schema with the following field names and data types:

      | **Field name**    | **Apache Iceberg data type**                                                                                                        |
      | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
      | `is_extracted`    | `varchar`                                                                                                                           |
      | `points`          | `varchar`                                                                                                                           |
      | `system`          | `varchar`                                                                                                                           |
      | `layout_width`    | `bigint`                                                                                                                            |
      | `layout_height`   | `bigint`                                                                                                                            |
      | `filetype`        | `varchar`                                                                                                                           |
      | `languages`       | `array(varchar)`                                                                                                                    |
      | `page_number`     | `varchar`                                                                                                                           |
      | `image_base64`    | `varchar`                                                                                                                           |
      | `image_mime_type` | `varchar`                                                                                                                           |
      | `filename`        | `varchar`                                                                                                                           |
      | `url`             | `varchar`                                                                                                                           |
      | `record_locator`  | `varchar`                                                                                                                           |
      | `entities`        | `row(items array(row(entity varchar, type varchar)), relationships array(row("from" varchar, relationship varchar, "to" varchar)))` |

      Incoming elements that do not have matching column
      names will be dropped upon record insertion. For example, if the incoming data has an element named `sent_from` and there is no
      column named `sent_from` in the table, the `sent_from` element will be dropped upon record insertion. You should modify the preceding
      sample table creation statement to add columns for any additional elements that you want to be included upon record
      insertion.

      To increase query performance, Iceberg uses [hidden partitioning](https://iceberg.apache.org/docs/latest/partitioning/) to
      group similar rows together when writing. You can also
      [explicitly define partitions](https://prestodb.io/docs/current/connector/iceberg.html#create-table) as part of the
      preceding `CREATE TABLE` statement.

* The name of the target namespace (also known as a schema) within the target catalog, and name of the target table within that schema. To get these:

  1. [Log in to your IBM Cloud account](https://cloud.ibm.com/login).
  2. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the
     top navigation bar.
  3. In the list of resources, expand **Databases**, and then click the target watsonx.data data store instance.
  4. Click **Open web console**.
  5. If prompted, log in to the web console.
  6. On the sidebar, click **Data manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the
     top navigation bar.
  7. On the **Browse data** tab, expand the name of the target catalog, and note the names of the target schema and target table.

* The name of the column in the target table that uniquely identifies each of the records in the table.

* To improve performance, the target table should be set to regularly remove old metadata files. To do this, run the following Python script.
  (You cannot use the preceding `CREATE TABLE` statement, or other SQL statements such as `ALTER TABLE`, to set this behavior.) To get the
  values for the specified environment variables, see the preceding instructions.

  ```python theme={null}
  # Improves performance by setting the target table to regularly remove 
  # old metadata files. 
  #
  # First, install the following dependencies into your Python virtual 
  # environment:
  # 
  # pip install requests pyiceberg pyarrow
  #
  # Then, set the following environment variables:
  #
  # IBM_IAM_API_KEY - An API key value for the target IBM Cloud account.
  # IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT - The metastore REST endpoint 
  #     value for the target Apache Iceberg catalog in the target IBM watsonx.data 
  #     data store instance.
  # IBM_COS_BUCKET_PUBLIC_ENDPOINT - The target IBM Cloud Object Storage (COS) 
  #     instance’s endpoint value.
  # IBM_COS_ACCESS_KEY - An HMAC access key ID for the target COS instance.
  # IBM_COS_SECRET_ACCESS_KEY - The associated HMAC secret access key ID for the 
  #     target HMAC access key.
  # IBM_COS_BUCKET_REGION - The target COS instance’s region short ID.
  # IBM_ICEBERG_CATALOG - The name of the target Iceberg catalog.
  # IBM_ICEBERG_SCHEMA - The name of the target namespace (also known as a schema) 
  #     in the target catalog.
  # IBM_ICEBERG_TABLE - The name of the target table in the target schema.
  #
  # To get these values, see the Unstructured documentation for the 
  #     IBM watsonx.data connector.

  import os
  import requests
  from pyiceberg.catalog import load_catalog

  def main():
     # Get a bearer token for the target IBM Cloud account.   
     bearer_token = requests.post(
        url="https://iam.cloud.ibm.com/identity/token",
        headers={
              "Content-Type": "application/x-www-form-urlencoded",
              "Accept": "application/json"
        },
        data={
              "grant_type": "urn:ibm:params:oauth:grant-type:apikey", 
              "apikey": os.getenv("IBM_IAM_API_KEY")
        }
     ).json().get("access_token")

     # Connect to the target Iceberg catalog.
     catalog = load_catalog(
        os.getenv("IBM_ICEBERG_CATALOG"),
        **{
              "type": "rest",
              "uri": f"https://{os.getenv("IBM_ICEBERG_CATALOG_METASTORE_REST_ENDPOINT")}/mds/iceberg",
              "token": bearer_token,
              "warehouse": os.getenv("IBM_ICEBERG_CATALOG"),
              "s3.endpoint": os.getenv("IBM_COS_BUCKET_PUBLIC_ENDPOINT"),
              "s3.access-key-id": os.getenv("IBM_COS_ACCESS_KEY"),
              "s3.secret-access-key": os.getenv("IBM_COS_SECRET_ACCESS_KEY"),
              "s3.region": os.getenv("IBM_COS_BUCKET_REGION")
        },
     )
              
     # Load the target table.
     table = catalog.load_table(f"{os.getenv("IBM_ICEBERG_SCHEMA")}.{os.getenv("IBM_ICEBERG_TABLE")}")

     # Set the target table's properties to remove old metadata files.
     with table.transaction() as transaction:
        transaction.set_properties(
              {
                 "commit.manifest.min-count-to-merge": 10,
                 "commit.manifest-merge.enabled": True,
                 "write.metadata.previous-versions-max": 10,
                 "write.metadata.delete-after-commit.enabled": True,
              }
        )

     # Confirm that the target table's properties were set as expected.
     print(table.metadata.properties)

  if __name__ == "__main__":
     main()
  ```

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **IBM watsonx.data**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Iceberg Endpoint** (*required*): The metastore REST endpoint for the target Apache Iceberg-based catalog within the IBM watsonx.data data store instance. Do not include `https://` in this value.
* **Object Storage Endpoint** (*required*): The public endpoint for the target bucket within the IBM Cloud Object Storage (COS) instance that is associated with the catalog. Do not include `https://` in this value.
* **Object Storage Region** (*required*): The region short ID (such as us-east) for the bucket.
* **IAM API Key** (*required*): A valid API key value for the IBM Cloud account.
* **Access Key ID** (*required*): A valid hash-based message authentication code (HMAC) access key ID for the COS instance.
* **Secret Access Key** (*required*): The HMAC secret access key for the access key ID.
* **Catalog** (*required*): The name of the target Apache Iceberg-based catalog within the IBM watsonx.data data store instance.
* **Namespace** (*required*): The name of the target namespace (also known as a schema) within the catalog.
* **Table** (*required*): The name of the target table within the namespace (schema).
* **Max Connection Retries**: The maximum number of retries when connecting to the catalog. Typically, an optimal setting is `15`. The default is `10`. If specified, it must be a number between `2` and `100`, inclusive.
* **Max Upload Retries**: The maximum number of retries when uploading data. Typically, an optimal setting is `150`. The default is `50`. If specified, it must be a number between `2` and `500`, inclusive.
* **Record ID Key**: The name of the column that uniquely identifies each record in the target table. The default is `record_id`.

## Learn more

* <Icon icon="blog" />  [Getting Started with Unstructured and IBM watsonx.data](https://unstructured.io/blog/getting-started-with-unstructured-and-ibm-watsonx-data).
* <Icon icon="blog" />  [Unstructured + IBM watsonx.data: A New OEM Partnership Powering the Future of Enterprise AI](https://unstructured.io/blog/unstructured-ibm-watsonx-data-a-new-oem-partnership-powering-the-future-of-enterprise-ai)


# Kafka
Source: https://docs.unstructured.io/pipelines/destinations/kafka



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Kafka.

The requirements are as follows.

* A Kafka cluster in [Confluent Cloud](https://www.confluent.io/confluent-cloud).
  ([Create a cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#create-ak-clusters).)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

  <iframe title="YouTube video player" />

* The [hostname and port number](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#view-a-ak-cluster) of the bootstrap Kafka cluster to connect to..

* The name of the topic to read messages from or write messages to on the cluster.
  [Create a topic](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).
  [Access available topics](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).

* For authentication, an [API key and secret](https://docs.confluent.io/cloud/current/security/authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.html#add-an-api-key).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Kafka**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bootstrap Server** (*required*): The hostname of the bootstrap Kafka cluster to connect to.
* **Port**: The port number of the cluster.
* **Group ID**: The ID of the consumer group, if any, that is associated with the target Kafka cluster.
  (A consumer group is a way to allow a pool of consumers to divide the consumption of data
  over topics and partitions.) The default is `default_group_id` if not otherwise specified.
* **Topic** (*required*): The unique name of the topic to read messages from and write messages to on the cluster.
* **Number of messages to consume**: The maximum number of messages to get from the topic. The default is `100` if not otherwise specified.
* **Batch Size**: The maximum number of messages to send in a single batch. The default is `100` if not otherwise specified.
* **API Key** (*required*): The Kafka API key value.
* **Secret** (*required*): The secret value for the Kafka API key.

## Learn more

* <Icon icon="blog" />  [Unstructured Platform Now Integrates with Apache Kafka in Confluent Cloud](https://unstructured.io/blog/unstructured-platform-now-integrates-with-apache-kafka-in-confluent-cloud)


# Milvus
Source: https://docs.unstructured.io/pipelines/destinations/milvus



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Milvus.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Milvus cloud-based instances (such as Milvus on IBM watsonx.data, or Zilliz Cloud) are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Milvus local and cloud-based instances are supported.

* For Milvus on IBM watsonx.data, you will need:

  <iframe title="YouTube video player" />

  * An [IBM Cloud account](https://cloud.ibm.com/registration).

  * An IBM watsonx.data [Lite plan](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-tutorial_prov_lite_1)
    or [Enterprise plan](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started_1) within your IBM Cloud account.

    * If you are provisioning a Lite plan, be sure to choose the **Generative AI** use case when prompted, as this is the only use case offered that includes Milvus.

  * A [Milvus service instance in IBM watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-adding-milvus-service).

    * If you are creating a Milvus service instance within a watsonx.data Lite plan, when you are prompted to choose a Milvus instance size, you can only select **Lite**. Because the Lite
      Milvus instance size is recommended only for 384 dimensions, you should also use an embedding model that uses 384 dimensions only.
    * If you are creating a Milvus service instance within a watsonx.data Enterprise plan, you can choose any available Milvus instance size. However, all Milvus instance sizes other than
      **Custom** are recommended only for 384 dimensions, which means you should use an embedding model that uses 384 dimensions only.
      The **Custom** Milvus instance size is recommended for any number of dimensions.

  * The URI of the instance, which takes the format of `https://`, followed by the instance's **GRPC host**, followed by a colon and the **GRPC port**.
    This takes the format of `https://<host>:<port>`. To get this information, do the following:

    a. Sign in to your IBM Cloud account.<br />
    b. On the sidebar, click the **Resource list** icon. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the title bar.<br />
    c. Expand **Databases**, and then click the name of the target **watsonx.data** plan.<br />
    d. Click **Open web console**.<br />
    e. On the sidebar, click **Infrastructure manager**. If the sidebar is not visible, click the **Global navigation** icon to the far left of the title bar.<br />
    f. Click the target Milvus service instance.<br />
    g. On the **Details** tab, under **Type**, click **View connect details**.<br />
    h. Under **Service details**, expand **GRPC**, and note the value of **GRPC host** and **GRPC port**.<br />

  * The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.

  * The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database. Note the collection requirements at the end of this section.

  * The username and password to access the instance.

    * The username for Milvus on IBM watsonx.data is typically `ibmlhapikey`.

      <Note>
        More recent versions of Milvus on IBM watsonx.data require `ibmlhapikey_<your-IBMid>` instead, where `<your-IBMid>` is
        your IBMid, for example `me@example.com`. To get your IBMid, do the following:

        1. Sign in to your IBM Cloud account.
        2. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
        3. In the sidebar, expand **Manage identities**, and then click **Users**.
        4. In the list of users, click your user name.
        5. On the **User details** tab, in the **Details** tile, note the value of **IBMid**.
      </Note>

    * The password for Milvus on IBM watsonx.data is in the form of an IBM Cloud user API key. To create an IBM Cloud user API key:

      a. Sign in to your IBM Cloud account.<br />
      b. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.<br />
      c. On the sidebar, under **Manage identities**, click **API keys**. If the sidebar is not visible, click the **Navigation Menu** icon to the far left of the title bar.<br />
      d. Click **Create**.<br />
      e. Enter some **Name** for the API key.<br />
      f. Optionally, enter some **Description** for the API key.<br />
      g. For **Leaked action**, leave **Disable the leaked key** selected.<br />
      h. For **Session management**, leave **No** selected.<br />
      i. Click **Create**.<br />
      j. Click **Download** (or **Copy**), and then download the API key to a secure location (or paste the copied API key into a secure location). You won't be able to access this API key from this dialog again. If you lose this API key, you can create a new one (and you should then delete the old one).<br />

* For Zilliz Cloud, you will need:

  <iframe title="YouTube video player" />

  * A [Zilliz Cloud account](https://cloud.zilliz.com/signup).

  * A [Zilliz Cloud cluster](https://docs.zilliz.com/docs/create-cluster).

  * The URI of the cluster, also known as the cluster's *public endpoint*, which takes a format such as
    `https://<cluster-id>.<cluster-type>.<cloud-provider>-<region>.cloud.zilliz.com`. To get this public endpoint value, do the following:

    1. After you sign in to your Zilliz Cloud account, on the sidebar, in the list of available projects, select the project that contains the cluster.
    2. On the sidebar, click **Clusters**.
    3. Click the tile for the cluster.
    4. On the **Cluster Details** tab, on the **Connect** subtab, copy the **Public Endpoint** value.

  * The username and password to access the cluster, as follows:

    1. After you sign in to your Zilliz Cloud account, on the sidebar, in the list of available projects, select the project that contains the cluster.
    2. On the sidebar, click **Clusters**.
    3. Click the tile for the cluster.
    4. On the **Users** tab, copy the name of the user.
    5. Next to the user's name, under **Actions**, click the ellipsis (three dots) icon, and then click **Reset Password**.
    6. Enter a new password for the user, and then click **Confirm**. Copy this new password.

  * The name of the [database](https://docs.zilliz.com/docs/database#create-database) in the instance.

  * The name of the [collection](https://docs.zilliz.com/docs/manage-collections-console#create-collection) in the database.

    The collection must have a defined schema before Unstructured can write to the collection. The minimum viable
    schema for Unstructured contains only the fields `element_id`, `embeddings`, `record_id`, and `text`, as follows.
    `type` is an optional field, but highly recommended. For settings for additional Unstructured-produced fields,
    such as the ones within `metadata`, see the usage notes toward the end of this section and adapt them to your specific needs.

    | Field Name                       | Field Type        | Max Length | Dimension |
    | -------------------------------- | ----------------- | ---------- | --------- |
    | `element_id` (primary key field) | **VARCHAR**       | `200`      | --        |
    | `embeddings` (vector field)      | **FLOAT\_VECTOR** | --         | `384`     |
    | `record_id`                      | **VARCHAR**       | `200`      | --        |
    | `text`                           | **VARCHAR**       | `65536`    | --        |
    | `type`                           | **VARCHAR**       | `200`      | --        |

    In the **Create Index** area for the collection, next to **Vector Fields**, click **Edit Index**. Make sure that for the
    `embeddings` field, the **Field Type** is set to **FLOAT\_VECTOR** and the **Metric Type** is set to **Cosine**.

    <Warning>
      The number of dimensions for the `embeddings` field must match the number of dimensions for the embedding model that you plan to use.
    </Warning>

    <Note>
      Fields with a **VARCHAR** data type are limited to a maximum length of 65,535 characters. Attempting to exceed this character count
      will cause Unstructured to throw errors when attempting to write to a Milvus collection, and the associated Unstructured job could fail.
      For example, `metadata` fields that typically exceed these character counts include `image_base64` and `orig_elements`.
    </Note>

    <Info>
      The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
    </Info>

* For Milvus local, you will need:

  * A [Milvus instance](https://milvus.io/docs/install-overview.md).
  * The [URI](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Client/MilvusClient.md) of the instance.
  * The name of the [database](https://milvus.io/docs/manage_databases.md) in the instance.
  * The name of the [collection](https://milvus.io/docs/manage-collections.md) in the database.
    Note the collection requirements at the end of this section.
  * The [username and password, or token](https://milvus.io/docs/authenticate.md) to access the instance.

All Milvus instances require the target collection to have a defined schema before Unstructured can write to the collection. The minimum viable
schema for Unstructured contains only the fields `element_id`, `embeddings`, `record_id`, and `text`, as follows.
`type` is an optional field, but highly recommended.

This example code demonstrates the use of the
[Python SDK for Milvus](https://pypi.org/project/pymilvus/) to create a collection with this schema,
targeting Milvus on IBM watsonx.data. For the `MilvusClient` arguments to connect to other types of Milvus deployments, see your Milvus provider's documentation:

```python Python theme={null}
import os

from pymilvus import (
    MilvusClient,
    FieldSchema,
    DataType,
    CollectionSchema
)

DATABASE_NAME   = "default"
COLLECTION_NAME = "my_collection"

client = MilvusClient(
    uri="https://" +
        os.getenv("MILVUS_USER") + 
        ":" + 
        os.getenv("MILVUS_PASSWORD") + 
        "@" + 
        os.getenv("MILVUS_GRPC_HOST") + 
        ":" + 
        os.getenv("MILVUS_GRPC_PORT"),
    db_name=DATABASE_NAME
)

# IMPORTANT: The number of dimensions for the "embeddings" field that
# follows must match the number of dimensions for the embedding model 
# that you plan to use.
fields = [
    FieldSchema(name="element_id", dtype=DataType.VARCHAR, is_primary=True, max_length=200),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=384),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=200),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="type", dtype=DataType.VARCHAR,max_length=200, nullable=True) # Optional, but highly recommended.
]

schema = CollectionSchema(fields=fields)

client.create_collection(
    collection_name=COLLECTION_NAME,
    schema=schema,
    using=DATABASE_NAME
)

index_params = client.prepare_index_params()

index_params.add_index(
    field_name="embeddings",
    metric_type="COSINE",
    index_type="IVF_FLAT",
    params={"nlist": 1024}
)

client.create_index(
    collection_name=COLLECTION_NAME,
    index_params=index_params
)

client.load_collection(collection_name=COLLECTION_NAME)
```

For objects in the `metadata` field that Unstructured produces and that you want to store in Milvus, you must create fields in your Milvus collection schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You could create corresponding fields in your Milvus collection schema by using the following field names, data types, and related settings for the
Python SDK for Milvus:

```python theme={null}
# ...
fields = [
    # Define minimum viable required fields: "element_id", "embeddings", "record_id", and "text".
    # "type" is an optional field, but highly recommended.
    FieldSchema(name="element_id", dtype=DataType.VARCHAR, is_primary=True, max_length=200),
    FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=1536),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=200),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="type",dtype=DataType.VARCHAR,max_length=200, nullable=True),
    # Define optional fields, such as these ones related to the "metadata" field.
    FieldSchema(name="is_extracted", dtype=DataType.VARCHAR, max_length=5, nullable=True),
    FieldSchema(name="coordinates_points", dtype=DataType.JSON, nullable=True),
    FieldSchema(name="coordinates_system", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="coordinates_layout_width", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="coordinates_layout_height", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="filetype", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="languages", dtype=DataType.ARRAY, element_type=DataType.VARCHAR, max_length=64, max_capacity=10, nullable=True),
    FieldSchema(name="page_number", dtype=DataType.INT32, nullable=True),
    FieldSchema(name="image_mime_type", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="filename", dtype=DataType.VARCHAR, max_length=256, nullable=True),
    FieldSchema(name="data_source_url", dtype=DataType.VARCHAR, max_length=1024, nullable=True),
    FieldSchema(name="data_source_record_locator_protocol", dtype=DataType.VARCHAR, max_length=64, nullable=True),
    FieldSchema(name="data_source_record_locator_remote_file_path", dtype=DataType.VARCHAR, max_length=1024, nullable=True),
    FieldSchema(name="entities_items", dtype=DataType.JSON, nullable=True),
    FieldSchema(name="entities_relationships", dtype=DataType.JSON, nullable=True)
]
# ...
```

<Note>
  Fields with a `DataType.VARCHAR` data type are limited to a maximum length of 65,535 characters. Attempting to exceed this character count
  will cause Unstructured to throw errors when attempting to write to a Milvus collection, and the associated Unstructured job could fail.
  For example, `metadata` fields that typically exceed these character counts include `image_base64` and `orig_elements`.
</Note>

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Milvus**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields for Milvus on IBM watsonx.data:

* **Name** (*required*): A unique name for this connector.

* **GRPC Host** (*required*): The GRPC host name for the Milvus instance.

* **GRPC Port**: The GRPC port number for the instance.

* **DB Name**: The name of the database in the instance. The default is `default` if not otherwise specified.

* **Collection Name** (*required*): The name of the collection in the database.

* **Username**: The username to access the Milvus instance. The default is `ibmlhapikey` if not otherwise specified.

  <Note>
    More recent versions of Milvus on IBM watsonx.data require `ibmlhapikey_<your-IBMid>` instead, where `<your-IBMid>` is
    your IBMid, for example `me@example.com`. To get your IBMid, do the following:

    1. Sign in to your IBM Cloud account.
    2. In the title bar, click **Manage** and then, under **Security and access**, click **Access (IAM)**.
    3. In the sidebar, expand **Manage identities**, and then click **Users**.
    4. In the list of users, click your user name.
    5. On the **User details** tab, in the **Details** tile, note the value of **IBMid**.
  </Note>

* **API Key** (*required*): The IBM Cloud user API key.

Fill in the following fields for Milvus on Zilliz Cloud:

* **Name** (*required*): A unique name for this connector.
* **URI** (*required*): The URI of the Milvus instance, for example: `https://12345.serverless.gcp-us-west1.cloud.zilliz.com`.
* **DB Name**: The name of the database in the instance. The default is `default` if not otherwise specified.
* **Collection Name** (*required*): The name of the collection in the database.
* **Username** (*required*): The username to access the Milvus instance.
* **Password** (*required*): The password corresponding to the username to access the instance.

Fill in the following fields for other Milvus deployments:

* **Name** (*required*): A unique name for this connector.
* **URI** (*required*): The URI of the Milvus instance, for example: `https://12345.serverless.gcp-us-west1.cloud.zilliz.com`.
* **DB Name**: The name of the database in the instance. The default is `default` if not otherwise specified.
* **Collection Name** (*required*): The name of the collection in the database.
* **Username** (*required*): The username to access the Milvus instance.
* **Password** (*required*): The password corresponding to the username to access the instance.


# MongoDB
Source: https://docs.unstructured.io/pipelines/destinations/mongodb



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to MongoDB.

The requirements are as follows.

<iframe title="YouTube video player" />

The MongoDB requirements for a MongoDB Atlas deployment include:

<Warning>
  For MongoDB Atlas, SCRAM-SHA-1 is not supported for authentication. This means that cluster types that only
  include SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, and **Serverless**, are **not** supported.
  Unstructured only supports SCRAM-SHA-256 for MongoDB Atlas, which is cryptographically stronger than SCRAM-SHA-1.

  If you try to test or use a connector that refers to a cluster type that only includes SCRAM-SHA-1, the
  operation will fail, and you will get an error message similar to the following:
  `[digital envelope routines] unsupported`.
</Warning>

* A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).

* A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). Be sure to **not**
  select a cluster type that only includes SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, or **Serverless**.

* The cluster must be reachable from your application environment, for example by adding IP addresses to your IP access list. [Learn more](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).

* The cluster must be configured to allow IP address. [Learn how](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries).

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

* The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).

* The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).

* The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).

  <Note>
    For the destination connector, Unstructured recommends that all documents in the target collection have a field
    named `record_id` with a `String` data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the collection or, in some cases, the operation could fail altogether.
  </Note>

* The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

  ```text theme={null}
  mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
  ```

  To get the connection string in MongoDB Atlas, do the following:

  1. Log in to your MongoDB Atlas console.
  2. In the sidebar, under **Databases**, click **Clusters**.
  3. Click on the cluster you want to connect to.
  4. Click **Connect**.
  5. Click **Drivers**.
  6. Under **Add your connection string into your application code**, copy the connection string.
     You can then close the **Connect** dialog in MongoDB Atlas.

     Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

  [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **MongoDB**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Database** (*required*): The name of the database on the instance.
* **Collection** (*required*): The name of the collection within the database.
* **Connection String** (*required*): The MongoDB instance connection string.

## Learn more

* <Icon icon="blog" />  [How to go from S3 to MongoDB with no code using Unstructured](https://unstructured.io/blog/how-to-go-from-s3-to-mongodb-with-no-code-using-unstructured)


# MotherDuck
Source: https://docs.unstructured.io/pipelines/destinations/motherduck



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to MotherDuck.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [MotherDuck account](https://app.motherduck.com).

* A [MotherDuck access token](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck/#creating-an-access-token) for the account.

* A database in the account.

  * [Create a database](https://motherduck.com/docs/sql-reference/motherduck-sql-reference/create-database/).
  * [List available databases](https://motherduck.com/docs/key-tasks/database-operations/basics-operations/#listing-databases).

  You can run commands to manage MotherDuck databases, schemas, tables, and more in the
  [MotherDuck UI](https://motherduck.com/docs/getting-started/motherduck-quick-tour/) or for example by connecting to MotherDuck with the
  [DuckDB CLI](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/connecting-to-motherduck/).

* A schema in the target database.

  * [Create a schema](https://duckdb.org/docs/sql/statements/create_schema.html).
  * You can list available schemas and their parent catalogs by running the following command in the MotherDuck UI or the DuckDB CLI:

    ```sql theme={null}
    SELECT * FROM information_schema.schemata;
    ```

  The MotherDuck connector uses the default schema name of `main` if not otherwise specified.

* A table in the target schema.

  * [Create a table](https://duckdb.org/docs/sql/statements/create_table).
  * You can list available tables in a schema by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog and schema names:

    ```sql theme={null}
    USE <catalog_name>.<schema_name>;
    SHOW TABLES;
    ```

  The MotherDuck connector uses the default table name of `elements` if not otherwise specified.

  For maximum compatibility, Unstructured recommends the following table schema:

  ```sql theme={null}
  CREATE TABLE elements (
      id VARCHAR,
      element_id VARCHAR,
      text TEXT,
      embeddings FLOAT[],
      type VARCHAR,
      system VARCHAR,
      layout_width DECIMAL,
      layout_height DECIMAL,
      points TEXT,
      url TEXT,
      version VARCHAR,
      date_created INTEGER,
      date_modified INTEGER,
      date_processed DOUBLE,
      permissions_data TEXT,
      record_locator TEXT,
      category_depth INTEGER,
      parent_id VARCHAR,
      attached_filename VARCHAR,
      filetype VARCHAR,
      last_modified TIMESTAMP,
      file_directory VARCHAR,
      filename VARCHAR,
      languages VARCHAR[],
      page_number VARCHAR,
      links TEXT,
      page_name VARCHAR,
      link_urls VARCHAR[],
      link_texts VARCHAR[],
      sent_from VARCHAR[],
      sent_to VARCHAR[],
      subject VARCHAR,
      section VARCHAR,
      header_footer_type VARCHAR,
      emphasized_text_contents VARCHAR[],
      emphasized_text_tags VARCHAR[],
      text_as_html TEXT,
      regex_metadata TEXT,
      detection_class_prob DECIMAL,
      partitioner_type VARCHAR
  );
  ```

  You can list the schema of a table by running the following commands in the MotherDuck UI or the DuckDB CLI, replacing the target catalog, schema, and table names:

  ```sql theme={null}
  USE <catalog_name>.<schema_name>;
  DESCRIBE TABLE <table_name>;
  ```

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **MotherDuck**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Database** (*required*): The name of the target MotherDuck database.
* **Schema** (*required*): The name of the target schema within the database.
* **Table**: The name of the target table within the schema. By default, this table is named `elements` if not otherwise specified.
* **MotherDuck Token** (*required*): The access token value within the MotherDuck account that has the appropriate access to the target database, schema, and table.

## Learn more

* <Icon icon="blog" />  [Unstructured's New MotherDuck Integration](https://unstructured.io/blog/unstructured-s-new-motherduck-integration)


# Neo4j
Source: https://docs.unstructured.io/pipelines/destinations/neo4j



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Neo4j.

The requirements are as follows.

* A [Neo4j deployment](https://neo4j.com/deployment-center/).

  * For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local Neo4j deployments are not supported.
  * For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local Neo4j deployments are supported.

  The following video shows how to set up a Neo4j Aura deployment:

  <iframe title="YouTube video player" />

* The username and password for the user who has access to the Neo4j deployment. The default user is typically `neo4j`.

  * For a Neo4j Aura instance, the defaut user's is typically set when the instance is created.
  * For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, the default user is typically set during the deployment process.
  * For a local Neo4j deployment, you can [set the default user's initial password](https://neo4j.com/docs/operations-manual/current/configuration/set-initial-password/) or [recover an admin user and its password](https://neo4j.com/docs/operations-manual/current/authentication-authorization/password-and-user-recovery/).

* The connection URI for the Neo4j deployment, which starts with `neo4j://`, `neo4j+s://`, `bolt://`, or `bolt+s://`; followed by `localhost` or the host name; and sometimes ending with a colon and the port number (such as `:7687`). For example:

  * For a Neo4j Aura deployment, browse to the target Neo4j instance in the Neo4j Aura account and click **Connect > Drivers** to get the connection URI, which follows the format `neo4j+s://<host-name>`. A port number is not used or needed.
  * For an AWS Marketplace, Microsoft Azure Marketplace, or Google Cloud Marketplace deployment of Neo4j, see
    [Neo4j on AWS](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-aws/),
    [Neo4j on Azure](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-azure/), or
    [Neo4j on GCP](https://neo4j.com/docs/operations-manual/current/cloud-deployments/neo4j-gcp/)
    for details about how to get the connection URI.
  * For a local Neo4j deployment, the URI is typically `bolt://localhost:7687`
  * For other Neo4j deployment types, see the deployment provider's documentation.

  [Learn more](https://neo4j.com/docs/browser-manual/current/operations/dbms-connection).

* The name of the target database in the Neo4j deployment. A default Neo4j deployment typically contains two standard databases: one named `neo4j` for user data and another
  named `system` for system data and metadata. Some Neo4j deployment types support more than these two databases per deployment;
  Neo4j Aura instances do not.

  * [Create additional databases](https://neo4j.com/docs/operations-manual/current/database-administration/standard-databases/create-databases/)
    for a local Neo4j deployment that uses Enterprise Edition; or for Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP deployments.
  * [Get a list of additional available databases](https://neo4j.com/docs/operations-manual/current/database-administration/standard-databases/listing-databases/)
    for a local Neo4j deployment that uses Enterprise Edition; or for Neo4j on AWS, Neo4j on Azure, or Neo4j on GCP deployments.

## Graph Output

The graph output of the Neo4j destination connector is represented in the following diagram:

```mermaid theme={null}
graph BT
    subgraph dn [Document Node]
    D[Document]
    end
    style dn stroke-dasharray: 5
    
    subgraph en [Element Nodes]
    UE1[UnstructuredElement]
    UE2[UnstructuredElement]
    UE3[UnstructuredElement]
    UE4[UnstructuredElement]
    UE5[UnstructuredElement]
    UE6[UnstructuredElement]
    end
    style en stroke-dasharray: 5
    
    UE1 -->|PART_OF_DOCUMENT| D
    UE2 -->|PART_OF_DOCUMENT| D
    UE3 -->|PART_OF_DOCUMENT| D
    UE4 -->|PART_OF_DOCUMENT| D
    UE5 -->|PART_OF_DOCUMENT| D
    UE6 -->|PART_OF_DOCUMENT| D

    subgraph cn [Chunk Nodes]
    C1[Chunk]
    C2[Chunk]
    C3[Chunk]
    C4[Chunk]
    end
    style cn stroke-dasharray: 5
    
    C1 -->|NEXT_CHUNK| C2
    C2 -->|NEXT_CHUNK| C3
    C3 -->|NEXT_CHUNK| C4

    C1 -->|PART_OF_DOCUMENT| D
    C2 -->|PART_OF_DOCUMENT| D
    C3 -->|PART_OF_DOCUMENT| D
    C4 -->|PART_OF_DOCUMENT| D

    UE1 -.->|PART_OF_CHUNK| C1
    UE2 -.->|PART_OF_CHUNK| C1
    UE3 -.->|PART_OF_CHUNK| C2
    UE4 -.->|PART_OF_CHUNK| C3
    UE5 -.->|PART_OF_CHUNK| C4
    UE6 -.->|PART_OF_CHUNK| C4
```

[View the preceding diagram in full-screen mode](https://mermaid.live/view#pako:eNqFlN9vgjAQx_-Vps-6REEfeFiyFZYli7hskCyTxXS0ihFaU9oHo_7vq_IjgIzyxN330157d70TjDmh0IFbgQ8JeA4iBvSXq9_CQRhYuTxWGWUS-Br9KQC39pYOyki5VB5Tel2XS8H3dExwnmAh8NEBs4LohKA6hJfSOkJe7hh6k1XI9C4qlkpQUjK1Oh1UrUHVHlRng-p8QO1kgRqzoC8JxuPH8_vTR7BevqzdJQoXnh-cgVvf0wRYJsA2ATMTMP8f6FQz1tVEiWL7Vi3RpHBW5rRtWm3TbpmdnMbGnKIipb73FazRa-i_nXXAKvC9ZFWHuJfs6nrIUCVkKBIy1AjZpgTfGuWhwVRnnDT6ZFC3-vVpo0v6dKvRJH263eiRXh2OYEZFhndEj5nTlY6gTPSriaCjfwndYJXKCEbsolGsJP88shg6-onRERRcbRPobHCaa0sdCJbU3WHdbFmFHDD75jyrIUp2kotFMddu4-3yB3k-fcg).

In the preceding diagram:

* The `Document` node represents the source file.
* The `UnstructuredElement` nodes represent the source file's Unstructured `Element` objects, before chunking.
* The `Chunk` nodes represent the source file's Unstructured `Element` objects, after chunking.
* Each `UnstructuredElement` node has a `PART_OF_DOCUMENT` relationship with the `Document` node.
* Each `Chunk` node also has a `PART_OF_DOCUMENT` relationship with the `Document` node.
* Each `UnstructuredElement` node has a `PART_OF_CHUNK` relationship with a `Chunk` element.
* Each `Chunk` node, except for the "last" `Chunk` node, has a `NEXT_CHUNK` relationship with its "next" `Chunk` node.

Learn more about [document elements](/concepts/document-elements) and [chunking](/concepts/chunking).

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

```text theme={null}
MATCH path=(source)-[relationship]->(target)
RETURN path
```

Query for `Chunk` to `Document` relationships:

```text theme={null}
MATCH (chunk:Chunk)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN chunk, relationship, doc
```

Query for `UnstructuredElement` to `Document` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, relationship, doc
```

Query for `UnstructuredElement` to `Chunk` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
RETURN element, relationship, chunk
```

Query for `Chunk` to `Chunk` relationships:

```text theme={null}
MATCH (this:Chunk)-[relationship:NEXT_CHUNK]->(previous:Chunk)
RETURN this, relationship, previous
```

Query for `UnstructuredElement` to `Chunk` to `Document` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[ecrelationship:PART_OF_CHUNK]-(chunk:Chunk)-[cdrelationship:PART_OF_DOCUMENT]->(doc:Document)
RETURN element, ecrelationship, chunk, cdrelationship, doc
```

Query for `UnstructuredElements` containing the text `jury`, and show their `Chunk` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE element.text =~ '(?i).*jury.*'
RETURN element, relationship, chunk
```

Query for the `Chunk` with the specified `id`, and show its `UnstructuredElement` relationships:

```text theme={null}
MATCH (element:UnstructuredElement)-[relationship:PART_OF_CHUNK]->(chunk:Chunk)
WHERE chunk.id = '731508bf53637ce4431fe93f6028ebdf'
RETURN element, relationship, chunk
```

Additionally, for the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API's workflow operations](/api-reference/workflow/),
when a [Named entity recognition (NER)](/concepts/enriching/ner) DAG node is added to a custom workflow,
any recognized entities are output as `Entity` nodes in the graph.

This additional graph output of the Neo4j destination connector is represented in the following diagram:

```mermaid theme={null}
graph TD
    Chunk -->|HAS_ENTITY| Entity
    Entity -->|ENTITY_TYPE| Entity
```

In the preceding diagram:

* The `Chunk` node represents one of the source file's Unstructured `Element` objects, after chunking.
* The `Entity` node represents a recognized entity.
* A `Chunk` node can have `HAS_ENTITY` relationships with `Entity` nodes.
* An `Entity` node can have `ENTITY_TYPE` relationships with other `Entity` nodes.

Some related example Neo4j graph queries include the following.

Query for all available nodes and relationships:

```text theme={null}
MATCH path=(source)-[relationship]->(target)
RETURN path
```

Query for `Entity` to `Entity` relationships:

```text theme={null}
MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
RETURN child, relationship, parent
```

Query for `Entity` nodes containing the text `PERSON`, and show their `Entity` relationships:

```text theme={null}
MATCH (child:Entity)-[relationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id = 'PERSON'
RETURN child, relationship, parent
```

Query for `Entity` nodes containing the text `amendment`, and show their `Chunk` relationships:

```text theme={null}
MATCH (element:Chunk)-[relationship:HAS_ENTITY]->(entity:Entity)
WHERE entity.id =~ '(?i).*amendment.*'
RETURN element, relationship, entity
```

QUERY FOR `Entity` nodes containing the text `PERSON`, and show their `Entity` to `Entity` to `Chunk` relationships:

```text theme={null}
MATCH (chunk:Chunk)-[ccrelationship:HAS_ENTITY]-(child:Entity)-[cprelationship:ENTITY_TYPE]->(parent:Entity)
WHERE parent.id =~ 'PERSON'
RETURN chunk, ccrelationship, child, cprelationship, parent
```

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Neo4j**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **URI** (*required*): The connection URI for the Neo4j deployment, which typically starts with `neo4j://`, `neo4j+s://`, `bolt://`, or `bolt+s://`;
  is followed by the host name; and ends with a colon and the port number (such as `:7473`, `:7474`, or `:7687`).
* **Database** (*required*): The name of the target database in the Neo4j deployment. A default Neo4j deployment typically contains a standard database named `neo4j` for user data.
* **Username** (*required*): The name of the user who has access to the Neo4j deployment. A default Neo4j deployment typically contains a default user named `neo4j`.
* **Password** (*required*): The password for the user.
* **Batch Size**: The maximum number of nodes or relationships to be transmitted per batch. The default is `100` if not otherwise specified.


# OneDrive
Source: https://docs.unstructured.io/pipelines/destinations/onedrive



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to OneDrive.

The requirements are as follows.

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* The OneDrive and SharePoint Online plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.

  The following video shows how to get a path:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials, and a username and password. Both authentication types require a Microsoft Entra ID app registration.

  <Note>
    The OneDrive connector does not support any other authentication methods, such as multifactor (MFA) or passwordless authentication.
  </Note>

  You will need to provide the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration that has access to the target OneDrive account, and
  the app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Files.ReadWrite.All` (if both reading and writing are needed)
  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
  * `Directory.Read.All`

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* Both authentication types also require the User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

* For username and password authentication, you will also need the password for the target UPN.

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **OneDrive**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Client ID** (*required*): The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account.
* **Tenant ID** (*required*): The directory (tenant) ID of the Entra ID app registration.
* **Authority URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
* **Principal Name** (*required*): The User Principal Name (UPN) for the OneDrive user account in Entra ID. This is typically the user's email address.
* **Client Credential** (*required*): The client secret for the Entra ID app registration.
* **Password** (*required* for username and password authentication): The password for the target UPN.
* **Path** (source connector only): The path to the target folder in the OneDrive account, starting with the account's root folder, for example `my-folder/my-subfolder`.
* **Recursive** (source connector only): Check this box to recursively access files from subfolders within the specified OneDrive path.
* **Remote URL** (destination connector only): `onedrive://`, followed by the path to the target folder in the OneDrive account, starting with the account's root folder, for example `onedrive://my-folder/my-subfolder`.


# OpenSearch
Source: https://docs.unstructured.io/pipelines/destinations/opensearch



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to OpenSearch.

## Set up OpenSearch

Supported OpenSearch installations vary by product:

* **[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):** Non-local OpenSearch only. Local OpenSearch instances are not supported.
* **[Unstructured Ingest](/open-source/ingestion/overview):** Both local and non-local OpenSearch instances are supported.

### Set up an AWS OpenSearch Service domain

To set up an [AWS OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) domain, complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.

4. Click **Create domain**.

5. In the **Name** tile, for **Domain name**, enter some unique domain name for your new OpenSearch domain.

6. In the **Domain creation method** tile, select **Easy create**. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With **Standard create**, you must enable FGAC manually.) [Standard create method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html).

7. In the **Engine options** tile, for **Version**, AWS recommends that you select the latest version.

8. In the **Network** tile, for **Network**, select a network access method.
   For faster setup, this example uses the **Public access** method.
   [VPC access method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#prerequisites-vpc-endpoints).

9. For **IP address type**, select **Dual-stack mode**.

10. In the **Fine-grained access control** (FGAC) tile, do one of the following:

    * If you want to use an existing AWS IAM user in the AWS account as the domain's master user, then for **Master user**, select **Set IAM ARN as master user**. Then enter the IAM ARN for the master user in the **IAM ARN** box.
    * If you want to create a master user and password as the domain's master user instead, then for **Master user**, select **Create master user**. Then specify some username and password for this
      new master user by filling in the **Master username**, **Master password**, and **Confirm master password** fields. Make
      sure to save the master user's password in a secure location.

11. Click **Create**.

12. After the domain is created, you must allow Unstructured to access the domain, as follows:

    a. If the new domain's settings page is not already showing, open it as follows:
    in your Amazon OpenSearch Service console, on the sidebar, expand **Managed clusters**, and then click **Domains**. Then,
    in the list of available domains, click the name of the newly created domain.<br />
    b. On the **Security configuration** tab, click **Edit**.<br />
    c. In the **Access policy** tile, for **Domain access policy**, select **Only use fine-grained access control**.<br />
    d. Click **Clear policy**. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).<br />
    e. Click **Save changes**.

### Set up an Amazon OpenSearch Serverless collection

To set up an [Amazon OpenSearch Serverless collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-create-console.html), complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Serverless**, and then click **Dashboard**.

4. Click **Create collection**.

5. In the **Collection details** tile, for **Collection name**, enter some unique name for your new OpenSearch Serverless collection.
   Optionally, for **Description**, enter some meaningful description for your new collection.

6. For **Collection type**, select **Search**.

   <Note>
     Unstructured does not support the **Vector search** collection type. If you need vector search support, you can either continue
     with these steps to use the **Search** collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead.
     However, note that the Amazon OpenSearch Serverless **Search** collection type is not as optimal as the **Vector search** collection type.
   </Note>

7. In the **Collection creation method** tile, select **Standard create**.

8. For **Encryption**, choose an AWS KMS key type.

9. For **Network access settings**, choose an **Access type**.

10. For **Resource type**, select both **Enable access to OpenSearch endpoint** and **Enable access to OpenSearch Dashboards**.

11. Click **Next**.

12. In the **Definition method** tile, select **JSON**.

13. In the **JSON editor** box, enter the following JSON, replacing the following placeholders:

    * Replace `<collection-name>` with the name of the new OpenSearch Serverless collection.
    * Replace `<account-id>` with the target AWS account ID.
    * Replace `<user-id>` with the ID of the target AWS IAM user.

    ```json theme={null}
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    ```

14. Click **Next**.

15. For **Data access policy settings**, select **Create as a new data access policy**.

16. In the **Name and description** tile, enter some unique name and an optional description for the new data access policy.

17. Click **Next**.

18. Enter any desired index details, and click **Next** again. For example:

    a. For **Index name**, enter the name of the new index in the collection.<br />
    b. For **Automatic Semantic Enrichment fields**, click **Add**, enter `embeddings` for **Automatic Semantic Enrichment field name**, click **Add**, and click **Confirm**.<br />
    c. For **Lexical search fields**, click **Add**, enter `text` for **Field name** and select **Text** for **Data type**, click **Add**, and click **Confirm**.<br />

19. Click **Submit**.

### Set up a local OpenSearch instance

The following video shows how to set up a [local OpenSearch](https://opensearch.org/downloads.html) instance.

<iframe title="YouTube video player" />

### Get the host URL

Find your host URL in the AWS console using the steps for your OpenSearch type.

#### Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.
4. In the list of available domains, click the name of your domain.
5. In the **General information** tile, copy the value of **Domain endpoint v2 (dual stack)**.

#### Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Serverless**, and then click **Dashboard**.
4. In the list of available collections, click the name of your collection.
5. On the **Overview** tab, in the **Endpoint** tile, copy the value of **OpenSearch endpoint**.

#### Local OpenSearch instance

Your local instance URL depends on how OpenSearch is installed and configured. For guidance, see [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) in the OpenSearch documentation.

### Create a search index

The name of the search index on the instance is required.

For the destination connector, if you need to create an index and you're using a master user and password as the domain's master user, you can use for example the following `curl` command. Replace the following placeholders:

* Replace `<host>` with the instance's host URL.
* Replace `<port>` with the instance's port number, which is typically `443` (for encrypted connections, and less commonly `9200` for unencrypted connections).
* Replace `<master-username>` with the master user's name, and replace `<master-password>` with the master user's password.
* Replace `<index-name>` with the name of the new search index on the instance.
* Replace `<index-schema>` with the schema for the new search index on the instance. A schema is optional; see the explanation
  following this `curl` command for more information.

```bash theme={null}
curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
```

If you're using an existing AWS IAM user as the domain's master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the preceding `curl` command. To learn how, see [create-index](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-index.html) in the AWS CLI Command Reference.

For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

For objects in the `metadata` field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index's schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
required by Unstructured whenever you create your own index schema:

* `element_id`
* `record_id`, which is required by Unstructured for intelligent record updates.
* `type`, which is not required, but highly recommended.
* `text`
* `embeddings` if embeddings are generated; make sure to set `dimension` to the same number of dimensions as the embedding model generates.

```json theme={null}
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

See also:

* [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/)
* [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
* [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
* [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
* [Unstructured document elements and metadata](/concepts/document-elements)

### Set up master user authentication

<Note>
  If you are using Enterprise Connect on a [dedicated instance](/business/dedicated-instances/overview), you do not need **master user credentials**. Skip to the **Enterprise Connect** section on this page.
</Note>

For non-local OpenSearch instances, or if you're using basic authentication to a local OpenSearch instance, you will need the master user's name and password.

For local OpenSearch instances, if you're using certificates for authentication instead of basic authentication, you will need:

* The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
* The path to the combined private key and certificate file, or
* The paths to the separate private key and certificate files.

To learn more, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an OpenSearch connector to use Enterprise Connect, first set up your AWS IAM role:

### Create the IAM role

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your OpenSearch resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

### Attach a permissions policy

To allow Unstructured to access your OpenSearch installation, you may need to attach a permissions policy to the IAM role. Whether it is required depends on your OpenSearch type and how your domain access policy is configured.

#### Amazon OpenSearch Service (managed)

A permissions policy is typically not required for a managed OpenSearch service domain that has the following:

* The domain's access policy is permissive (for example, `"Principal": "*"`)
* Fine-grained access control (FGAC) is the primary access control

Both conditions above are satisfied if you selected **Easy create** in step 6 (which enables FGAC by default) and followed step 12(d) (for the domain access policy) in the [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain) section. No identity-based permissions policy is needed, and the trust policy you attached in [Create the IAM role](#create-the-iam-role) is sufficient for Enterprise Connect to function. Skip to [Grant access to OpenSearch resources](#grant-access-to-opensearch-resources).

If either condition is not met and your domain's access policy explicitly restricts access to specific principals or actions, attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:<region>:<account-id>:domain/<domain-name>/*"
    }
  ]
}
```

| Placeholder     | Value                                                  |
| --------------- | ------------------------------------------------------ |
| `<region>`      | The AWS region of the domain, for example `us-east-1`. |
| `<account-id>`  | The 12-digit ID of your AWS account.                   |
| `<domain-name>` | The name of the OpenSearch domain.                     |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

#### Amazon OpenSearch Serverless

The OpenSearch Serverless connector requires a permissions policy. Attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "aoss:APIAccessAll",
      "Resource": "arn:aws:aoss:<region>:<account-id>:collection/<collection-id>"
    }
  ]
}
```

| Placeholder       | Value                                                      |
| ----------------- | ---------------------------------------------------------- |
| `<region>`        | The AWS region of the collection, for example `us-east-1`. |
| `<account-id>`    | The 12-digit ID of your AWS account.                       |
| `<collection-id>` | The ID of the OpenSearch Serverless collection.            |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

### Grant access to OpenSearch resources

After creating the IAM role and configuring AWS permissions, you must also authorize the role within OpenSearch itself. AWS authorization and OpenSearch authorization are separate systems.

The steps in this section configure the OpenSearch authorization for your IAM role. The process you follow depends on your OpenSearch type.

#### Amazon OpenSearch Service (managed) — Map the role inside OpenSearch

You only need to map the IAM role inside OpenSearch if fine-grained access control (FGAC) is enabled on your domain:

* If you selected **Easy create** in step 6 of [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain), FGAC was enabled automatically and these steps are required.
* If your domain has FGAC disabled, skip this section. Your IAM role setup is complete. Proceed to the next section to create your connector.

**Select or create an OpenSearch role**

An *OpenSearch role* is an internal security construct, distinct from an AWS IAM role. The IAM role ARN is mapped to an internal *OpenSearch role* by registering it as a **backend role**. This is how OpenSearch links an external AWS identity to an internal permission set. First, select or create the *OpenSearch role* that has permission to perform the actions your connector needs:

* For a quick proof-of-concept, use the built-in `all_access` *OpenSearch role*, already present in OpenSearch Dashboards under **Security** → **Roles**.
* For production, first [create a least-privilege *OpenSearch role*](https://opensearch.org/docs/latest/security/access-control/users-roles/) in OpenSearch Dashboards under **Security** → **Roles**, scoped to the specific indices your connector will access.

Once you have your *OpenSearch role*, complete the steps below to map your IAM role ARN to it. Skipping this mapping causes the connector to fail with a 403 error, even if the IAM role passes the AWS authorization checks:

```
security_exception ... no permissions for [indices:admin/aliases/get] and User [name=arn:aws:iam::...:role/...]
```

For more information about managing OpenSearch roles, see [Users and roles](https://opensearch.org/docs/latest/security/access-control/users-roles/) in the OpenSearch documentation.

**Map your IAM role ARN to the OpenSearch role**

You can add the mapping using OpenSearch Dashboards or the API:

* **OpenSearch Dashboards:** Go to **Security** → **Roles** → select the *OpenSearch role* → **Mapped users** tab → add the IAM role ARN under **Backend roles**. For more information, see [Fine-grained access control in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html) in the *AWS OpenSearch Service Developer Guide*.

* Using the **API**:

  ```bash theme={null}
  curl -XPUT "$DOMAIN/_plugins/_security/api/rolesmapping/<opensearch-role-name>" \
    -H "Content-Type: application/json" \
    -u "$ADMIN_USER:$ADMIN_PASS" \
    -d '{"backend_roles":["<assumed-role-arn>"]}'
  ```

  Replace `<opensearch-role-name>` with the name of the *OpenSearch role* that you selected or created in this section. Replace `<assumed-role-arn>` with the ARN of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role).

  <Note>
    `PUT` replaces the entire role mapping. To preserve existing backend roles, first `GET` the current mapping and include all existing entries in your `PUT` request.
  </Note>

#### Amazon OpenSearch Serverless — Add the role to a data access policy

OpenSearch Serverless does not use FGAC. Authorization is managed through data access policies attached to the collection.

<Note>
  OpenSearch Serverless limits each AWS account and region to 10 data access policies. Where possible, add the IAM role ARN to an existing policy rather than creating a new one.
</Note>

Use the AWS CLI to update the collection's data access policy:

```bash theme={null}
# 1. Find the data access policy for your collection
aws opensearchserverless list-access-policies \
  --type data \
  --region <region> \
  --resource "collection/<collection-name>"

# 2. Get the current policy and note the policyVersion value in the response
aws opensearchserverless get-access-policy \
  --type data \
  --name <policy-name> \
  --region <region>

# 3. Update the policy, adding the IAM role ARN to the Principal array
aws opensearchserverless update-access-policy \
  --type data \
  --name <policy-name> \
  --policy-version <version-from-step-2> \
  --region <region> \
  --policy file://policy.json
```

The `--policy-version` value comes from the `get-access-policy` command above (#2) and enables optimistic concurrency. The `--policy` document replaces the entire policy. Include all existing principals and rules, not just your addition.

Example `policy.json`, replacing the placeholder values:

```json theme={null}
[
  {
    "Rules": [
      {
        "Resource": ["collection/<collection-name>"],
        "Permission": ["aoss:DescribeCollectionItems"],
        "ResourceType": "collection"
      },
      {
        "Resource": ["index/<collection-name>/*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument",
          "aoss:UpdateIndex"
        ],
        "ResourceType": "index"
      }
    ],
    "Principal": ["arn:aws:iam::<account-id>:role/<role-name>"],
    "Description": "Unstructured connector access"
  }
]
```

| Placeholder         | Value                                                                                               |
| ------------------- | --------------------------------------------------------------------------------------------------- |
| `<collection-name>` | The name of the OpenSearch Serverless collection.                                                   |
| `<account-id>`      | The 12-digit ID of your AWS account.                                                                |
| `<role-name>`       | The name of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role). |

**For source-only connectors:** remove `aoss:WriteDocument`,  `aoss:UpdateIndex`, and `aoss:CreateIndex` from the `Permission` array.

<Note>
  Policy changes propagate within seconds, but may occasionally take up to a minute. If the connector returns a 403 immediately after updating the policy, wait a moment and retry before treating it as a configuration error.
</Note>

## Create the destination connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **OpenSearch**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host URL** (*required*): The OpenSearch instance's host URL, which typically takes the form of `https://<host>:<port>`.
* **Index Name** (*required*): The name of the search index on the instance.

If you're using **Basic Auth**, fill in the following fields:

* For **Authentication Method**, select **Basic Auth**.
* For **Username**, enter the master user's name.
* For **Password**, enter the master user's password.

If you're using **IAM Credentials**, fill in the following fields:

* For **Authentication Method**, select **IAM Credentials**.
* For **AWS Access Key ID**, enter the AWS access key ID for the AWS IAM user.
* For **AWS Secret Access Key**, enter the AWS secret access key for the corresponding AWS access key ID.

If you're using **IAM + Session Token**, fill in the following fields:

* For **Authentication Method**, select **IAM + Session Token**.
* For **AWS Access Key ID**, enter the temporary AWS access key ID for the AWS IAM user.
* For **AWS Secret Access Key**, enter the temporary AWS secret access key for the corresponding AWS access key ID.
* For **AWS STS Token**, enter the temporary AWS STS token for temporary access.

<Warning>
  If you are using the **IAM + Session Token** authentication method:<br /><br />
  AWS STS credentials (consisting of a temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding OpenSearch connector.<br /><br />
  You must get a new set of AWS STS credentials to replace the expired ones by, which produces a new, refreshed temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token. For more information, see [Request temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html).

  After you generate refreshed temporary AWS STS credentials, you must update the OpenSearch connector's settings with the new, refreshed AWS STS credentials.
</Warning>

If you're using **Enterprise Connect (AWS)** ([dedicated instance](/business/dedicated-instances/overview) only), fill in the following fields:

* For **Authentication Method**, select **Enterprise Connect (AWS)**.
* For **IAM Role ARN**, enter the ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
* For **External ID**, enter the unique value you chose when setting up your IAM role.


# Overview
Source: https://docs.unstructured.io/pipelines/destinations/overview



<Tip>
  If you do not have a destination location for Unstructured to send its processed data, skip over to the
  [Pinecone destination connector quickstart](/pipelines/destinations/pinecone-destination-quickstart). This quickstart
  guides you through the process of creating a free Pinecone account
  and creating a destination connector to connect Unstructured to a Pinecone dense serverless index within your Pinecone account.
</Tip>

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Destination connectors in Unstructured are designed to specify the endpoint for data processed within the platform. These connectors ensure that the transformed and analyzed data is securely and efficiently transferred to a storage system for future use, often to a vector database for tasks that involve high-speed retrieval and advanced data analytics operations.

<img alt="Destinations in the sidebar" />

To see your existing destination connectors, on the sidebar, click **Connectors**, and then click **Destinations**.

To create a destination connector:

1. In the sidebar, click **Connectors**.

2. Click **Destinations**.

3. Click **New** or **Create Connector**.

4. For **Name**, enter some unique name for this connector.

5. In the **Provider** area, click the destination location type that matches yours.

6. Click **Continue**.

7. Fill in the fields according to your connector type. To learn how, click your connector type in the following list:

   * [Astra DB](/pipelines/destinations/astradb)
   * [Azure AI Search](/pipelines/destinations/azure-ai-search)
   * [Azure Blob Storage](/pipelines/destinations/azure-blob-storage)
   * [Couchbase](/pipelines/destinations/couchbase)
   * [Databricks Volumes](/pipelines/destinations/databricks-volumes)
   * [Delta Tables in Amazon S3](/pipelines/destinations/delta-table)
   * [Delta Tables in Databricks](/pipelines/destinations/databricks-delta-table)
   * [Elasticsearch](/pipelines/destinations/elasticsearch)
   * [Google Cloud Storage](/pipelines/destinations/google-cloud)
   * [IBM FileNet](/pipelines/destinations/filenet)
   * [IBM watsonx.data](/pipelines/destinations/ibm-watsonxdata)
   * [Kafka](/pipelines/destinations/kafka)
   * [Milvus](/pipelines/destinations/milvus)
   * [MongoDB](/pipelines/destinations/mongodb)
   * [MotherDuck](/pipelines/destinations/motherduck)
   * [Neo4j](/pipelines/destinations/neo4j)
   * [OneDrive](/pipelines/destinations/onedrive)
   * [OpenSearch](/pipelines/destinations/opensearch)
   * [Pinecone](/pipelines/destinations/pinecone)
   * [PostgreSQL](/pipelines/destinations/postgresql)
   * [Qdrant](/pipelines/destinations/qdrant)
   * [Redis](/pipelines/destinations/redis)
   * [S3](/pipelines/destinations/s3)
   * [Snowflake](/pipelines/destinations/snowflake)
   * [Teradata](/pipelines/destinations/teradata-sql)
   * [Teradata Vector Store](pipelines/destinations/teradata-vector-v2)
   * [VAST](/pipelines/destinations/vast)
   * [Weaviate](/pipelines/destinations/weaviate)

8. If a **Continue** button appears, click it, and fill in any additional settings fields.

9. Click **Save and Test**.


# Pinecone
Source: https://docs.unstructured.io/pipelines/destinations/pinecone



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Pinecone.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Pinecone account. [Get an account](https://app.pinecone.io/).

* A Pinecone API key in your Pinecone account. To create an API key, do the following:

  1. [Sign in to your Pinecone account and open the API keys page](https://app.pinecone.io/organizations/-/projects/-/keys).
  2. Click **+ API key**.
  3. For **API key name**, enter some descriptive name for the API key.
  4. Click **Create key**.
  5. Copy the generated API key to some secure location. You will not be able to access this API key again after you close the dialog.

* A Pinecone serverless index in your Pinecone account.

  Creating a serverless index on your own is optional.
  An index is not required to exist in advance.

  When you set up the connector, at runtime, the index behavior is as follows:

  For the [Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):

  * Your workflow must contain an embedder node, and the embedder node must specify the embedding model that Unstructured will use to generate the embeedings.
  * If an existing index name is specified,
    and the number of dimensions that Unstructured generates does not match the number of dimensions that are specified in the existing index's embedding settings, the run will fail.
    You must change the number of dimensions in your workflow's embedder node or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The
    new index's name will be `u<short-workflow-id>-<short-embedding-model-name>-<number-of-dimensions>`.

  For [Unstructured Ingest](/open-source/ingestion/overview):

  * If an existing index name is specified, and Unstructured generates embeddings,
    but the number of dimensions that are generated does not match the existing index's embedding settings, the run will fail.
    You must change your Unstructured embedding settings or your existing index's embedding settings to match, and try the run again.
  * If an index name is not specified, Unstructured creates a new index in your Pinecone account. The new index's name will be `unstructuredautocreated`.

  <Note>
    If you create a new index or use an existing one, Unstructured recommends that all records in the target index have a field
    named `record_id` with a string data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the index or, in some cases, the operation could fail altogether.
  </Note>

  To create a serverless index on your own, do the following:

  1. [Sign in to your Pinecone account and open the Create a new index page](https://app.pinecone.io/organizations/-/projects/-/create-index/serverless).

  2. For **Enter index name**, enter some descriptive name for the index.

  3. For **Configuration**, select the check box labelled **Custom settings**, or click the tile labelled **Manual configuration**.

     <Warning>
       Do not click any of the other tiles, such as **text-embedding-3-large**. Clicking any of these other tiles will cause Pinecone to generate embeddings instead of
       having Unstructured generate them. If Pinecone generates embeddings instead of Unstructured, this could cause any related Unstructured workflows to fail.
     </Warning>

  4. For **Vector type**, select **Dense**.

  5. For **Dimension**, enter the number of dimensions for the embeddings that Unstructured will generate.

     <Warning>
       The number of dimensions that you enter here must match the number of dimensions for the embedding model that you use in any related Unstructured workflows or ingestion pipelines. If these numbers do not
       match in both places, this could cause any related Unstructured workflows or ingestion pipelines to fail.
     </Warning>

  6. For **Metric**, select **cosine**.

  7. Leave **Capacity mode** set to **Serverless**.

  8. You can leave **Cloud provider** and **Region** set to their default values, or you can select a cloud provider and region that is closest to you, if available.

  9. Click **Create index**.

* Within a Pinecone serverless index, custom [namespaces](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces) are supported but are not required.

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Pinecone**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Index Name**: The name of the index in the Pinecone database. If no value is provided, see the beginning of this article for the behavior at run time.
* **Namespace**: The name of any custom namespace in the Pinecone index to upsert data into. If no value is provided, the default namespace of `default` will be used.
* **Batch Size**: The number of records to use in a single batch. The default is `50` if not otherwise specified.
* **API Key** (*required*): The Pinecone API key.


# Pinecone destination connector quickstart
Source: https://docs.unstructured.io/pipelines/destinations/pinecone-destination-quickstart



Unstructured can connect to several types of [destinations](/pipelines/connectors#destinations). In this quickstart, you create a [Pinecone](https://www.pinecone.io) destination connector that you can add to your Unstructured [workflows](/pipelines/workflows).
This destination connector enables your Unstructured workflows to send processed data about your documents into a [Pinecone dense serverless index](https://docs.pinecone.io/guides/index-data/indexing-overview#dense-indexes) in your Pinecone account.

In this quickstart, you will:

* Create a free Pinecone account.
* Create a Pinecone destination connector in your Unstructured account.
* Add the Pinecone destination connector to a workflow in your Unstructured account.

If you are not able to complete the following steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

## Step 1: Create your Pinecone account

In this step, you create a [free Pinecone Starter account](https://www.pinecone.io/pricing/#plan-comparison-table). This account is limited to up to 2 users, up to 2 GB of total storage, and has rate-limited reads and writes.

If you already have a Pinecone account and want to use it instead, then skip ahead to Step 2.

1. Go to the Pinecone home page, at [https://www.pinecone.io](https://www.pinecone.io).

2. Click **Sign up**.

3. Enter your email address, and then click **Continue**.

4. Leaving this tab open in your web browser, go to your email inbox, and open the email messagefrom Pinecone. Copy the verification code that appears in this email message.

5. On the tab that you left open in your web browser, paste the verification code into the **your code** box, and then click **Continue**.

6. Select the radio button labelled **I'm building a small or personal project**, and then click **Start for free**.

7. On the next page, enter the following information to continue creating your Pinecone account:

   * Enter your **First Name** and **Last Name**.
   * For **Purpose of use**, select **Personal**.
   * For **Preferred coding language**, select **Python**.

8. Click **Continue**.

9. On the next page, enter the following information to finish creating your Pinecone account:

   * For **Use case**, select **RAG**.
   * For **What does your data look like**, select **I have raw documents**.
   * For **What is your goal**, select **Build an AI feature/product**.

10. Click **Get started**.

11. In the **API key generated** dialog that appears, click the copy icon to copy your API key's value. Save this key's value to a secure location, as you will need it later in Step 2.

    <img alt="Pinecone API key" />

    <Tip>
      If you lose this API key's value later, you will not be able to get it back. However, you can [create another API key](https://docs.pinecone.io/guides/projects/manage-api-keys#create-an-api-key).
      Be sure also to [delete the entry for the lost API key](https://docs.pinecone.io/guides/projects/manage-api-keys#delete-an-api-key).
    </Tip>

12. Click **Close**. The Pinecone user interface (UI) appears.

    <img alt="Pinecone user interface" />

## Step 2: Create the Pinecone destination connector

In this step, you create a Pinecone destination connector in your Unstructured account. This destination connector
is used by Unstructured to send processed data about your documents into a Pinecone dense serverless index in your Pinecone account.
Unstructured will create this index for you in your Pinecone account. You can then use this index later as input to your RAG applications
and agentic AI workflows.

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/pipelines/overview#how-am-i-billed%3F).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. On the sidebar, click **Connectors**.

   <img alt="Connectors sidebar icon" />

4. Click **+ New**.

5. Enter some unique name for this connector, for example `pinecone-destination-connector`.

6. For **Type**, click **Destination**.

7. For **Provider**, click **Pinecone**.

8. Click **Continue**.

   <img alt="Selecting the connector type" />

9. Leave **Index Name** blank.

10. Leave **Namespace** set to the default value of `default`.

11. Leave **Batch Size** set to the default value of `50`.

12. For **API Key**, enter the **API key** you noted in Step 1.

13. Click **Save and Test**, and wait while Unstructured tests the connector.

    <img alt="Entering connector settings" />

14. If a green **Successful** message appears, then you have successfully created the connector.

    <img alt="Successful connector test" />

    If, however, a red error message appears, fix the issue, and try this step again.

    If you cannot fix the issue, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

Congratulations! You have successfully created a Pinecone destination connector in your Unstructured account.

If you are not able to complete these steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

## Next steps

* If you do not have a source connector in your Unstructured account, then complete the [Dropbox source connector quickstart](/pipelines/sources/dropbox-source-quickstart).

  If you're not sure if you have a source connector, click **Connectors** in your Unstructured account's sidebar, and then click **Sources** to see if there are any listed.

  <img alt="Connectors sidebar icon" />

* If you already have a source connector, then you can add this Pinecone destination connector as well as your source connector to a workflow in your Unstructured account. To do this:

  1. Click **Workflows** in your Unstructured account's sidebar.

     <img alt="Workflows sidebar icon" />

  2. Click **New Workflow +**.

     <img alt="Begin creating a workflow" />

  3. With **Build it Myself** already selected, click **Continue**.

     <img alt="Build it myself workflow type" />

  4. Click the **Source** node. (Do not click **Drop file to test**.)

     <img alt="Workflow source node" />

  5. On the **Details** tab, click **Connectors**, and then click the name of your source connector.

     <img alt="Selecting the source connector" />

  6. Click the **Destination** node.

     <img alt="Workflow destination node" />

  7. On the **Details** tab, click the name of your Pinecone destination connector.

     <img alt="Selecting the destination connector" />

  8. Between the **Partitioner** and **Destination** nodes, click **+**.

     <img alt="Adding a workflow node" />

  9. Click **Transform > Embedder**.

     <img alt="Adding an embedder node" />

  10. On the **Details** tab, under **Azure OpenAI**, click the radio button labelled **text-embedding-3-small \[dim 1536]**. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

      <img alt="Selecting the embedding model" />

  11. Switch **Active** to on, and then click **Save**.

      <img alt="Making the workflow active" />

  12. Next to your workflow's name, click **Run**.

      <img alt="Running the workflow" />

  13. Click **Jobs** in your Unstructured account's sidebar.

      <img alt="Jobs sidebar icon" />

  14. Wait until the job shows **Finished** with a green checkmark.

      <img alt="Finished job status" />

  15. Log in to your Pinecone account, if you are not already logged in. To do this, go to [https://pinecone.io](https://pinecone.io), click
      **Log in**, and follow the on-screen instructions to finish logging in.

  16. On the sidebar, under **Database**, click **Indexes**.

      <img alt="View Pinecone indexes" />

  17. Click the name of the index that Unstructured created. The name should look like this: `u<random-number>-textembedding3-small-1536`.

      <img alt="Opening a Pinecone index" />

  18. On the **Browser** tab, click **List/Fetch**.

  19. For **Namespace**. select **default**.

  20. For **Operation**, select **List IDs**.

  21. For **Limit**, enter \`10.

  22. Click **List**.

      <img alt="Listing Pinecone index items" />

  23. To look at the data and metadata for any of the records that are in the list, click the ellipsis (three dots) icon, and then click **Fetch**.

      <img alt="Viewing Pinecone index item data" />

  24. When you are done looking at the record, click **Close**.

      <img alt="Closing a Pinecone index item" />

  25. To learn how to use this Pinecone index as input to your RAG applications and agentic AI workflows, see resources such as the following:

      * [Semantic search](https://docs.pinecone.io/guides/search/semantic-search) in the Pinecone documentation
      * [Filter by metadata](https://docs.pinecone.io/guides/search/filter-by-metadata) in the Pinecone documentation
      * [Rerank results](https://docs.pinecone.io/guides/search/rerank-results) in the Pinecone documentation
        \= [Build a RAG chatbot](https://docs.pinecone.io/guides/get-started/build-a-rag-chatbot) in the Pinecone documentation
      * [Pinecone integration with LangChain](https://python.langchain.com/docs/integrations/vectorstores/pinecone/) on the LangChain website

If you are not able to complete these steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# PostgreSQL
Source: https://docs.unstructured.io/pipelines/destinations/postgresql



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to PostgreSQL.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local PostgreSQL installations are not supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local PostgreSQL installations are supported.

The following video shows how to set up [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):

<iframe title="YouTube video player" />

The following video shows how to set up [Azure Database for PostgreSQL](https://azure.microsoft.com/products/postgresql):

<iframe title="YouTube video player" />

* A PostgreSQL instance.

  * [Create an Amazon RDS for PostgreSQL instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html).
  * [Create an Azure Database for PostgreSQL server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-deploy-on-azure-free-account).
  * [Install PostgreSQL locally](https://www.postgresql.org/docs/current/tutorial-install.html).

* The host name and port number for the instance.

  * For Amazon RDS for PostgreSQL, learn how to [get the host name and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html#postgresql-endpoint).
  * For Azure Database for PostgreSQL, learn how to [get the host](https://learn.microsoft.com/azure/postgresql/flexible-server/quickstart-create-server#get-the-connection-information). The port number is `5432`.
  * For local PostgreSQL installations, these values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.
  * For other installation types, see your PostgreSQL provider's documentation.

* Depending on your network security requirements, you might need to allow access to your instance only from specific IP addresses.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

  To learn how to allow these IP address ranges, see your PostgreSQL provider's documentation, for example with
  [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) or
  [Azure Database for PostgreSQL](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-manage-firewall-portal#create-a-firewall-rule-after-server-is-created).

  <Note>
    For Amazon RDS for PostgreSQL, Amazon recommends that you set the instance's **Public access** setting to **No** by default, as this
    approach is more secure. This means that no
    resources can connect to the instance outside of the instance's associated Virtual Private Cloud (VPC) without extra configuration.
    [Learn more](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Hiding).
    [Access an Amazon RDS instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html).

    If you must enable public access, set the instance's **Public access** setting to **Yes**, and then adjust the instance's related
    security group to allow this access.
    [Learn how](https://repost.aws/en/questions/QUxemKa9u5TV6CmLiO-r5prg/lost-public-access-to-aws-rds-postgresql-instance).

    [Troubleshoot issues with connecting to Amazon RDS instances](https://repost.aws/knowledge-center/rds-connectivity-instance-subnet-vpc).
  </Note>

* A database in the instance.

  * For Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, the default database name is `postgres` unless a custom database name was specified during the instance creation process.
  * For local PostgreSQL installations, learn how to [create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* A table in the database. Learn how to [create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

  For the destination connector, the table must have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the fields `id`, `element_id`, `record_id`, `text`, (and `embeddings`, if you are using `pgvector` and generating vector embeddings), as follows.
  `type` is an optional field, but highly recommended.

  If you are using `pgvector` and generating vector embeddings, the number of dimensions (in this example, `1536`) must match the number of dimensions for the associated embedding model that you use in any related Unstructured workflows or ingestion pipelines.

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT
    );
    ```

    ```sql PostgreSQL with pgvector theme={null}
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536)
    );
    ```
  </CodeGroup>

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  For objects in the `metadata` field that Unstructured produces and that you want to store in PostgreSQL, you must create fields in your table's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```

    ```sql PostgreSQL with pgvector  theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "embeddings" is required if you are generating vector embeddings.
    --   If you are generating embeddings, the number of dimensions in "embeddings" 
    --   must match the number of dimensions for the associated embedding model 
    --   that you use in any related Unstructured workflows or ingestion pipelines.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536),
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```
  </CodeGroup>

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  See also:

  * [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
  * [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  The following video shows how to use the `psql` utility to connect to PostgreSQL, list databases, and list and create tables:

  <iframe title="YouTube video player" />

* A user in the database, and a password for the user.

  * For Amazon RDS for PostgreSQL, learn how to [create a user](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Roles.html).
  * For Azure Database for PostgreSQL, learn how to [create a user](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-create-users).
  * For local PostgreSQL installations, learn how to [create a user](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* Database access for the user.

  * For Amazon RDS for PostgreSQL, learn how to [control user access](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Access.html).
  * For Azure Database for PostgreSQL, learn how to [control user access](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For local PostgreSQL installations, learn how to [give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).
  * For other installation types, see your PostgreSQL provider's documentation.

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **PostgreSQL**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host** (*required*): The host name of the target PostgreSQL instance.
* **DB Name** (*required*): The name of the target database on the instance.
* **Port** (*required*): The port number on the instance.
* **Username** (*required*): The name of the target user with the appropriate access to the instance.
* **Password** (*required*): The password for the user.
* **Table Name** (*required*): The name of the target table in the database.
* **Batch Size**: The maximum number of rows to transmit at a time. The default is `100` if not otherwise specified.
* **ID Column** (*required*, source connector only): The name of the ID column in the table.
* **Columns** (*required*, source connector only): A comma separated list of column names to process (including the specified **ID Column** column). The default is all columns if not otherwise specified.

## Learn more

* <Icon icon="blog" />  [PostgreSQL Integration in the Unstructured Platform](https://unstructured.io/blog/postgresql-integration-in-the-unstructured-platform)


# Qdrant
Source: https://docs.unstructured.io/pipelines/destinations/qdrant



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Qdrant.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only [Qdrant Cloud](https://qdrant.tech/documentation/cloud-intro/) is supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Qdrant Cloud,
  [Qdrant local](https://github.com/qdrant/qdrant), and [Qdrant client-server](https://qdrant.tech/documentation/quickstart/) are supported.

The following video shows how to set up Qdrant Cloud:

<iframe title="YouTube video player" />

* For Qdrant local, the path to the local Qdrant installation, for example: `/qdrant/local`

* For Qdrant client-server, the Qdrant server URL, for example: `http://localhost:6333`

* For Qdrant Cloud:

  * A [Qdrant account](https://cloud.qdrant.io/login).

  * A [Qdrant cluster](https://qdrant.tech/documentation/cloud/create-cluster/).

  * The cluster's URL. To get this URL, do the following:

    1. Sign in to your Qdrant Cloud account.
    2. On the sidebar, under **Dashboard**, click **Clusters**.
    3. Click the cluster's name.
    4. Note the value of the **Endpoint** field, for example: `https://<random-guid>.<region-id>.<cloud-provider>.cloud.qdrant.io`.

  * A [Qdrant API key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys).

* The name of the target [collection](https://qdrant.tech/documentation/concepts/collections) on the Qdrant local installation,
  Qdrant server, or Qdrant Cloud cluster.

  Qdrant requires the target collection to exist before Unstructured can write to the collection.
  The following example code demonstrates the use of the [Python Qdrant Client](https://pypi.org/project/qdrant-client/) to create
  a collection on a Qdrant Cloud cluster, configuring the collection for vectors with 3072 dimensions:

  ```python Python theme={null}
  from qdrant_client import QdrantClient, models
  import os

  client = QdrantClient(
      url=os.getenv("QDRANT_URL"),
      api_key=os.getenv("QDRANT_API_KEY")
  )

  client.create_collection(
      collection_name=os.getenv("QDRANT_COLLECTION"),
      vectors_config=models.VectorParams(
          size=3072,
          distance=models.Distance.COSINE
      )
  )

  collection = client.get_collection(
                   collection_name=os.getenv("QDRANT_COLLECTION")
               )

  print(f"The collection named '{os.getenv("QDRANT_COLLECTION")}' exists and " +
        f"has a status of '{collection.status}'.")
  ```

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Qdrant**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **URL** (*required*): The target Qdrant cluster's URL.
* **Collection Name** (*required*): The name of the target collection on the cluster.
* **Batch Size** The maximum number of records to be transmitted per batch. The default is `50` if not otherwise specified.
* **API Key** (*required*): The Qdrant API key value.

## Learn more

* <Icon icon="blog" />  [Qdrant Integration in the Unstructured Platform](https://unstructured.io/blog/streamlining-rag-pipeline-development-qdrant-integration-in-the-unstructured-platform)
* <Icon icon="blog" />  [How to go from S3 to Qdrant with no code using Unstructured](https://unstructured.io/blog/how-to-go-from-s3-to-qdrant-with-no-code-using-unstructured)


# Redis
Source: https://docs.unstructured.io/pipelines/destinations/redis



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Redis.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Redis](https://redis.io) database, for example in [Redis Cloud](https://redis.io/cloud/).
* The target database's hostname and port number. [Create a database in Redis Cloud](https://redis.io/docs/latest/operate/rc/rc-quickstart/#create-an-account).
* The username and password for the target database. [Get the username and password in Redis Cloud](https://redis.io/docs/latest/operate/rc/rc-quickstart/#connect-to-a-database).
* The database number for the target database. Redis databases are typically numbered from 0 to 15, with the default database number typically being 0.

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Redis**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **URI**: If **URI** is selected for **Authentication Method**, the connection URI for the target database user and password, hostname, and port number. Use the format of `redis://<username>:<password>@<hostname>:<port-number>`. If SSL encryption is enabled for the database, use `rediss://` instead of `redis://`.
* **Host** (*required* if **Password** is selected for **Authentication Method**): The target database's hostname.
* **Port**: If **Password** is selected for **Authentication Method**, the database's port number. The default is `6379` unless otherwise specified.
* **Username** (*required* if **Password** is selected for **Authentication Method**): The name of the database user (not the logged in user).
* **Password** (*required* if **Password** is selected for **Authentication Method**): The database user's password.
* **Database Index**: The number (index) for the target database. The default is `0` unless otherwise specified.
* **SSL**: Check this box to enable SSL encryption for the connection. Check this box only if SSL is already set up and enabled for the target database. This box is checked by default.
* **Key Prefix**: A string to prepend to each element's `element_id`. This is useful for distinguishing between different data sets in the same Redis instance. The default is an empty string.
* **Batch Size**: The maximum number of records to upload in a single batch. The default is `100` unless otherwise specified.

## Learn more

* <Icon icon="blog" />  [Redis Support in Unstructured Platform: Supercharging Your RAG Pipeline](https://unstructured.io/blog/redis-support-in-unstructured-platform-supercharging-your-rag-pipeline)
* <Icon icon="blog" />  [Getting Started with Unstructured and Redis](https://unstructured.io/blog/getting-started-with-unstructured-and-redis)


# S3
Source: https://docs.unstructured.io/pipelines/destinations/s3



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Amazon S3.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Amazon S3 requirements:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
  [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

  <img alt="Enable anonymous bucket access" />

* For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
  can be granted in one of the following ways:

  * Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
    To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
    Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

    <iframe title="YouTube video player" />

  * Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
    Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
    [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
    [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
    a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

    <Warning>
      AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
      generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
      You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
      a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

      To overwrite the expired credentials with the new set:

      * For Unstructured Pipelines, manually update the AWS Key, AWS Secret Key, and STS Token fields in Unstructured Pipelines
        for the corresponding S3 [source](/pipelines/sources/s3) or [destination](/pipelines/destinations/s3) connector.
      * For the Unstructured API, use the Unstructured API's workflow operations to call the
        [update source](/api-reference/api/source/update-source) or
        [update destination](/api-reference/api/destination/update-destination) connector operation
        for the corresponding S3 [source](/api-reference/workflow/sources/s3) or
        [destination](/api-reference/workflow/destinations/s3) connector.
      * For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
        corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
    </Warning>

* If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## FIPS and ambient credentials

<Note>
  The following information applies to [Unstructured Business dedicated instance and in-VPC](/business/overview) deployments only.
</Note>

Unstructured Business accounts support the Federal Information Processing Standard (FIPS) for
Amazon S3. [Learn more about AWS support for FIPS](https://aws.amazon.com/compliance/fips/). Specifically,
when creating an S3 connector with the [Unstructured Pipelines](/pipelines/overview) or
[Unstructured API's workflow operations](/api-reference/workflow/overview), Unstructured Business dedicated instance and in-VPC deployments
support the use of `fips`-based
[S3 endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html) and, for authenticated access to
S3 FIPS buckets, S3 ambient credentials.

To use Unstructured Pipelines to set up an S3 source or destination connector
to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.
2. When creating the connector, for the S3 connector's **Bucket URI** field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).
3. For the **Authentication Method** field, select **Ambient Credentials**.
4. Check the box labeled **Use Ambient Credentials**.
5. Save and test the connector.

To use the Unstructured API's workflow operations to set up an S3 source or
destination connector to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.

2. When creating the connector,
   for the `config` parameter's `remote_url` field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).

3. For the `config` parameter, add an `ambient_credentials` field, and set its value to true. For example:

   <CodeGroup>
     ```python Python SDK theme={null}
     # ...
     config={
         "...": "...",
         "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
         "ambient_credentials": True
     }
     # ...
     ```

     ```bash curl theme={null}
     # ...
     {
         "...": "...", 
         "config": {
             "...": "...",
             "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
             "ambient_credentials": "true"
         }
     }
     # ...
     ```
   </CodeGroup>

4. Run your code to create the connector.

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an S3 connector to use Enterprise Connect, first set up your AWS IAM role:

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your S3 resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

3. Attach a permissions policy to the role that grants access to your S3 bucket:

   * **Source connector:** at minimum `s3:GetObject`, `s3:ListBucket`
   * **Destination connector:** at minimum `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`

   Scope the policy to the specific bucket or prefix your connector will access.

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

Then, create the connector by following the steps in the **Create the  connector** section on this page.

## Create the destination connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Amazon S3**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bucket URI** (*required*): The URI for the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.
* **Recursive** (source connector only): Check this box to access subfolders within the bucket.

If you're using **Anonymous**, fill in the following field:

* For **Authentication Method**, select **Anonymous**.

If you're using **Secret**, fill in the following fields:

* For **Authentication Method**, select **Secret**.
* For **AWS Key**, enter the AWS access key ID for the authenticated AWS IAM user.
* For **AWS Secret Key**, enter the AWS secret access key corresponding to the preceding AWS access key ID.

If you're using **Enterprise Connect (AWS)** ([dedicated instance](/business/dedicated-instances/overview) only), fill in the following fields:

* For **Authentication Method**, select **Enterprise Connect (AWS)**.
* For **IAM Role ARN**, enter the ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
* For **External ID**, enter the unique value you chose when setting up your IAM role.

For all authentication methods, you can also fill in the following field:

* For **Custom URL**, enter a custom endpoint URL if connecting to a non-AWS S3 bucket.


# Snowflake
Source: https://docs.unstructured.io/pipelines/destinations/snowflake



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Snowflake.

The requirements are as follows.

* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.

  <iframe title="YouTube video player" />

  To get the identifier for the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
  3. On the **Account** tab, note the value of the **Account Identifier** field.

  Alternatively, the following Snowflake query returns the current account's identifier:

  ```text theme={null}
  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  ```

* A Snowflake user, which can be a service user (recommended) or a human user.

  To create a service user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Projects > Worksheets**.

  3. Click the **+** button to create a SQL worksheet.

  4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:

     * Replace `<service-user-name>` with some name for the service user.
     * Replace `<default-role-name>` with the name of any default role for the service user to use.

     ```sql theme={null}
     CREATE USER <service-user-name>
     DEFAULT_ROLE = "<default-role-name>"
     TYPE = SERVICE
     ```

  5. Click the arrow icon to run the worksheet, which creates the service user.

  6. To get their login name, on the navigation menu, click **Admin > Users & Roles**.

  7. On the **Users** tab, in the list of available users, click the name of the target user.

  8. In the **About** tile, note the **Login Name** for the user.

  To create a human user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & roles**.
  3. Click the **Users** tab.
  4. Click **+ User**.
  5. Follow the on-screen guidance to specify the user's settings.
  6. Click **Create User**.
  7. To get their login name, on the navigation menu, click **Admin > Users & Roles**.
  8. On the **Users** tab, in the list of available users, click the name of the target user.
  9. In the **About** tile, note the **Login Name** for the user.

* A programmatic access token (PAT) for the Snowflake user.

  <iframe title="YouTube video player" />

  To create a programmatic access token (PAT) for a user:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.

  3. On the **Users** tab, in the list of available users, click the name of the target user.

  4. In the **Programmatic access tokens** tile, click the **Generate new token** button.

  5. Follow the on-screen guidance to specify the PAT's settings.

     <Warning>
       You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later.
       Once this PAT expires, the connector will stop working.
       To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and
       update your connector's settings with your new PAT's value.

       Unstructured does not notify you when a PAT is about to expire or has already expired.
       You are responsible for tracking your PATs' expiration dates and taking corrective action before they expire.
     </Warning>

  6. Click **Generate**.

  7. Copy the generated PAT's value to a secure location, as you will not be able to access it again. If you lose this PAT's value,
     you will need to repeat this procedure to generate a new, replacement one.

  The PAT will not work unless the Snowflake account also has a valid
  [network rule](https://docs.snowflake.com/en/user-guide/network-rules) along with a valid
  [network policy](https://docs.snowflake.com/en/user-guide/network-policies) attached to that rule.
  The network rule must also be activated on the Snowflake account to begin taking effect.

  To create a valid network rule:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Rules**.

  3. Click **+ Network Rule**.

  4. Enter some name for the network rule.

  5. For **Type**, select **IPv4**.

  6. For **Mode**, select **Ingress**.

  7. For **Identifiers**, next to the magnifying glass icon, enter `0.0.0.0/0`, and then press **Enter**.

     <Note>
       The `0.0.0.0/0` value allows all IP addresses to access the Snowflake account.
       You can specify a more specific IP address range if you prefer. However, this more specific IP address range
       will apply to all users, including the user for which you created the PAT.
     </Note>

  8. Click **Create Network Rule**.

  To create a valid network policy, attaching the preceding network rule to this policy at the same time:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click **+ Network Policy**.
  4. Enter some name for the network policy.
  5. Make sure **Allowed** is selected.
  6. In the **Select rule** drop-down list, select the precedingnetwork rule to attach to this network policy.
  7. Click **Create Network Policy**.

  To activate the network rule in the account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click the name of the precedingnetwork policy to activate.
  4. In the policy's side panel, click the ellipsis (three dots) icon, and then click **Activate On Account**.
  5. Click **Activate policy**.

* (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake [user's login name (not username) and the user's password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
  This user must be a human user. Passwords are not supported for service users.

* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  <iframe title="YouTube video player" />

  * To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
  * To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
  * To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
  * To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
  * To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
  3. Click the **Roles** tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  ```text theme={null}
  SHOW ROLES;
  ```

  [Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).

* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

  <iframe title="YouTube video player" />

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:

  ```text theme={null}
  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  ```

* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

  <iframe title="YouTube video player" />

  To view a list of available databases in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  ```text theme={null}
  SHOW DATABASES;
  ```

* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  ```text theme={null}
  SHOW SCHEMAS;
  ```

  The following Snowflake query returns a list of available schemas for the database named `<database_name>` in the current account:

  ```text theme={null}
  SHOW SCHEMAS IN DATABASE <database_name>;
  ```

* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand **Tables**.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named `<schema_name>` in the datbase named
  `<database_name>` in the current account:

  ```text theme={null}
  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  ```

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the columns `ID`, `ELEMENT_ID`, and `RECORD_ID`. The columns `TEXT` and `TYPE` are optional, but highly recommended.
  If you are generating embeddings, then you must also include the column `EMBEDDINGS`.

  In the following `CREATE TABLE` statement, replace the following placeholders with the appropriate values:

  * `<database_name>`: The name of the target database in the Snowflake account.
  * `<schema_name>`: The name of the target schema in the database.
  * `<number-of-dimensions>`: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search,
    this value must match the number of dimensions that you plan to have Snowflake generate or search against.

  ```sql SQL theme={null}
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      ELEMENT_ID VARCHAR,
      RECORD_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>)
  );
  ```

  For objects in the `metadata` field that Unstructured produces and that you want to store in a Snowflake table, you must create columns in your table's schema that
  follow Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  ```sql theme={null}
  -- The ID, RECORD_ID, and ELEMENT_ID columns are required.
  -- TEXT and TYPE are not required but highly recommended.
  -- EMBEDDINGS is required if embeddings are being generated.
  -- All other "metadata" columns are optional.
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      RECORD_ID VARCHAR,
      ELEMENT_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
      IS_EXTRACTED VARCHAR,
      POINTS VARCHAR,
      SYSTEM VARCHAR,
      LAYOUT_WIDTH INTEGER,
      LAYOUT_HEIGHT INTEGER,
      FILETYPE VARCHAR,
      LANGUAGES ARRAY,
      PAGE_NUMBER VARCHAR,
      IMAGE_MIME_TYPE VARCHAR,
      FILENAME VARCHAR,
      URL VARCHAR,
      RECORD_LOCATOR VARCHAR
  );
  ```

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Snowflake**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Account ID** (*required*): The target Snowflake account's identifier.

* **Role** (*required*): The name of the Snowflake role that the user belongs to. This role must have the appropriate access to the target Snowflake warehouse, database, schema, and table.

* **User** (*required*): The target Snowflake user's login name (not their username).

* **Password** (*required*): The user's programmatic access token (PAT).

  <Note>
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  </Note>

* **Host** (*required*): The hostname of the target Snowflake warehouse.

* **Port** (*required*): The warehouse's port number. The default is `443` if not otherwise specified.

* **Database** (*required*): The name of the target Snowflake database.

* **Schema** (*required*): The name of the target Snowflake schema within the database.

* **Table** (*required* for source connector only): The name of the target Snowflake table within the database's schema. For the destination connector, the default is `elements` if not otherwise specified.

* **Columns** (source connector only): A comma-separated list of columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* **ID Column** (*required*, source connector only): The name of the column that uniquely identifies each record in the table.

* **Record ID Key** (destination connector only): The name of the column that uniquely identifies each record in the table. The default is `record_id` if not otherwise specified.

* **Batch Size** (*required*): The maximum number of rows to fetch for each batch. The default is `50` if not otherwise specified.

## Learn more

* <Icon icon="blog" />  [Getting Started with Unstructured and Snowflake](https://unstructured.io/blog/getting-started-with-unstructured-and-snowflake).


# Teradata
Source: https://docs.unstructured.io/pipelines/destinations/teradata-sql



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Teradata.

The requirements are as follows.

* A Teradata Vantage system that can be accessed by its host name or IP address.

  For example, a Teradata Vantage system in Teradata ClearScape Analytics Experience includes:

  * A Teradata ClearScape Analytics Experience account.
  * An environment in the account.
  * A Teradata Vantage database in the environment.
  * The name and password for a Teradata user who has the appropriate access to the database.

  [Learn how to create these in Teradata ClearScape Analytics Experience](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/).

* The system's corresponding host name or IP address.

  For example, you can get these values from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics Experience account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Host** value.<br />

* The name of the target database in the system. To get a list of available databases in the system, you can run a Teradata SQL query such as the following:

  ```sql theme={null}
  SELECT DatabaseName 
  FROM DBC.DatabasesV 
  ORDER BY DatabaseName;
  ```

* The name of the target table in the database. To get a list of available tables in a database, you can run a Teradata SQL query such as the following, replacing `<database-name>` with the name of the target database:

  ```sql theme={null}
  SELECT TableName
  FROM DBC.TablesV
  WHERE DatabaseName = '<database-name>' AND TableKind = 'T'
  ORDER BY TableName;
  ```

  When Unstructured writes rows to a table, the table's columns must have a schema that is compatible with Unstructured.
  Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances.
  This is because these schemas will vary based on
  your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
  any custom post-processing code that you run; and other factors.

  In any case, note the following about table schemas:

  * The following columns are always required by Unstructured: `record_id` and `element_id`.
  * The following columns are optional for Unstructured, but highly recommended: `text` and `type`.
  * The rest of the columns are optional and typically will be output by Unstructured as part of the `metadata` field.
  * If Unstructured is generating vector embeddings, the `embeddings` column is also required.

    <Warning>
      The destination connector outputs Unstructured-generated [embeddings](/concepts/embedding) that are not directly compatible
      with Teradata Enterprise Vector Store. To use embeddings with Teradata Enterprise Vector Store, Unstructured
      recommends that you choose from among the following options:

      * Define a column in your target table named `embeddings` that is of type `VARCHAR(64000)`, to store the
        Unstructured-generated embeddings. After Unstructured adds its embeddings to your `embeddings` column,
        choose from among Teradata's options to convert the `embeddings` column's `VARCHAR` values to the
        Teradata [VECTOR Data Type](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/VECTOR-Data-Type)
        yourself.
      * Omit any columns named `embedding`, `message`, or `num_tokens` from your target table. Then choose from
        among Teradata's options (such as [AI\_TextEmbeddings](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/In-Database-Analytic-Functions-for-Vector-Store))
        to have Teradata generate the embeddings for you, instead of having Unstructured generate them.
    </Warning>

  Here is an example table schema that is compatible with Unstructured. It includes all of the required and recommended columns, as
  well as a few additional columns that are typically output by Unstructured as part of the `metadata` field. Be sure to replace
  `<database-name>` with the name of the target database and `<table-name>` with the name of the target table (by Unstructured convention,
  the table name is typically `elements`, but this is not a requirement).

  ```sql theme={null}
  CREATE SET TABLE "<database-name>"."<table-name>" (
    "id" VARCHAR(64) NOT NULL,
    PRIMARY KEY ("id"),
    "record_id" VARCHAR(64),
    "element_id" VARCHAR(64),
    "text" VARCHAR(32000) CHARACTER SET UNICODE,
    "type" VARCHAR(50),
    "embeddings" VARCHAR(64000), -- Add this column only if Unstructured is generating vector embeddings.
    "last_modified" VARCHAR(50),
    "languages" VARCHAR(200),
    "file_directory" VARCHAR(500),
    "filename" VARCHAR(255),
    "filetype" VARCHAR(50),
    "record_locator" VARCHAR(1000),
    "date_created" VARCHAR(50),
    "date_modified" VARCHAR(50),
    "date_processed" VARCHAR(50),
    "permissions_data" VARCHAR(1000),
    "filesize_bytes" INTEGER,
    "parent_id" VARCHAR(64)
  )
  ```

* For the source connector, the name of the primary key column in the table (for example, a column named `id`, typically defined as `"id" VARCHAR(64) NOT NULL, PRIMARY KEY ("id")`).

* For the source connector, the names of any specific columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* For the destination connector, the name of the column in the table that uniquely identifies each record for Unstructured to perform any necessary record updates. By default convention, Unstructured expects this field to be named `record_id`.

* The name of the Teradata user who has the appropriate access to the target database.

  For example, you can get this from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Username** value.<br />

  The Teradata SQL command to get a list of available users is as follows:

  ```sql theme={null}
  SELECT UserName 
  FROM DBC.UsersV 
  ORDER BY UserName;
  ```

* The password for the user, which was set up when the user was created.

  If the user has forgotten their password, the Teradata SQL command to change a user's password is as follows, replacing `<user-name>` with the name of the user and `<new-password>` with the new password:

  ```sql theme={null}
  MODIFY USER <user-name> SET PASSWORD = '<new-password>';
  ```

  To change a user's password, you must be an administrator (such as the `DBC` user or another user with `DROP USER` privileges).

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Teradata**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host** (*required*): The hostname or IP address associated with the target Teradata Vantage database.
* **Database**: The name of the target database. By default, the default database name is used if not otherwise specified. To get the name of the default database, you can run the Teradata SQL command `SELECT DATABASE;`.
* **Table Name** (*required*): The name of the target table in the database.
* **Batch Size**: The maximum number of rows per batch. The default is `100` if not otherwise specified.
* **ID Column** (*required*, source connector only): The name of the primary key column that Teradata uses to uniquely identify each record in the table.
* **Record ID Key** (destination connector only): The name of the column that Unstructured uses to uniquely identify each record in the table for record update purposes. The default is `record_id` if not otherwise specified.
* **Columns** (source connector only): The names of the columns to fetch from the table, seperated by commas. By default, all columns are fetched unless otherwise specified.
* **Username** (*required*): The name of the user who has the appropriate access to the database.
* **Password** (*required*): The password for the user.


# Teradata Enterprise Vector Store
Source: https://docs.unstructured.io/pipelines/destinations/teradata-vector-v2

Send processed data from Unstructured to [Teradata Enterprise Vector Store](https://www.teradata.com/).

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

[Teradata Enterprise Vector Store](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Introduction-to-the-Enterprise-Vector-Store-User-Guide) is a specialized database designed to store and manage high-dimensional vectors (numeric arrays that encode meaning or similarity). Rather than rely on exact keyword matches, vector stores use semantic search by storing data as vectors.

## Requirements

You will need:

* The base URL for the Teradata Analytics REST API endpoint.

  The format depends on your **VantageCloud Enterprise** deployment type:

  * For private VPC: `https://<site-id>.private.cloud.teradata.com`
  * For public site gateway: `https://<site-id>.gateway.cloud.teradata.com`

  Replace `<site-id>` with your Teradata site ID. Contact your Teradata system administrator if you do not know your site ID.

  For more information, see [Vector Store Service Access and Authentication](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Setting-up-Vector-Store/Vector-Store-Service-Access-and-Authentication) in the *Teradata Enterprise Vector Store User Guide*.

  <Note>
    The REST API URL is not available for VantageCloud Lake (VCL) deployments.
    VCL deployments use JWT or PAT token-based authentication and are not currently supported by this connector.
  </Note>

* The username and password for your Teradata database account.

  This connector uses Basic Authentication (TD2), supported on AI-on-premises and VantageCloud Enterprise deployments.

  The account must have been granted the `TD_AIAdmin` role. This role is required to create vector collections. Contact your Teradata system administrator to confirm that your account has been granted this role.

  If you do not know your credentials, contact your Teradata system administrator.

  For more information, see [Authenticating Users](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Administering-Vector-Store/Authenticating-Users) in the *Teradata Enterprise Vector Store User Guide*.

* The name of the collection to create or add data to.

  A *vector collection* refers to an instance of Vector Store. An enterprise vector store may comprise thousands of vector collections, each tailored for a specific use case or workflow.

  Collection names must be unique within the vector store. They can only contain letters, numbers, and underscores, matching the following regex pattern: `^[a-zA-Z0-9_]+$`.

  * To use an existing collection, contact your Teradata system administrator for the collection name, or use the [Collection Manager](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Functional-Reference-of-V2-APIs-Vector-Collections/Using-Collection-Manager) to list available collections.
  * To create a new collection, provide a unique name. The connector creates the collection automatically on first use.

  For more information, see [Creating a Vector Collection](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Functional-Reference-of-V2-APIs-Vector-Collections/Creating-a-Vector-Collection) in the *Teradata Enterprise Vector Store User Guide*.

## Create the connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Teradata Vector Store**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Base URL** (*required*): The base URL for the Teradata Analytics REST API endpoint. The format depends on your **VantageCloud Enterprise** deployment type:

  * For private VPC: `https://<site-id>.private.cloud.teradata.com`
  * For public site gateway: `https://<site-id>.gateway.cloud.teradata.com`

  Replace `<site-id>` with your Teradata site ID. Contact your Teradata system administrator if you do not know your site ID.

  For more information, see [Vector Store Service Access and Authentication](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Setting-up-Vector-Store/Vector-Store-Service-Access-and-Authentication) in the *Teradata Enterprise Vector Store User Guide*.
* **User** (*required*):   The username for your Teradata database account.

  The account must have been granted the `TD_AIAdmin` role. This role is required to create vector collections. Contact your Teradata system administrator to confirm that your account has been granted this role.
* **Password** (*required*): The password for your Teradata database account.
* **Host**:  The database host. Defaults to the base URL if not otherwise specified.
* **Database**: The name of the database.
* **Collection Name** (*required*):   The name of the collection to create or add data to.

  Collection names must be unique within the vector store. Names can only contain letters, numbers, and underscores, matching the following regex pattern: `^[a-zA-Z0-9_]+$`.

  To create a new collection, provide a unique name. The connector creates the collection automatically on first use.
* **Use Precomputed Embeddings**: Check the **Use Precomputed Embeddings** box if your Unstructured workflow is configured to generate embedding vectors. Uncheck to have Teradata generate embeddings automatically.
* **Deduplicate**: Check the **Deduplicate** box to delete existing documents with the same filename before uploading new versions. This enables idempotent re-runs without destroying the entire collection. Uncheck to preserve existing documents.


# VAST
Source: https://docs.unstructured.io/pipelines/destinations/vast

Send processed data from Unstructured to [VAST](https://www.vastdata.com/).

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* A [VAST](https://www.vastdata.com/) user account with permissions to the bucket to which you want to connect.

* A bucket that contains the objects to which you want to connect. A bucket is a storage resource that uses Amazon's Simple Storage Service (S3) as an access protocol.

  To create a bucket, see [Creating S3 Buckets](https://kb.vastdata.com/documentation/docs/creating-s3-buckets-via-vms-1) in the *VAST Cluster Administrator's Guide*.

* The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/` (for example, `vast://my-bucket/`). If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/` (for example, `vast://my-bucket/my-folder/`).

* The URL for the VAST region in which the bucket is located (for example, `https://vast-cluster.my-company.com`).

* An enabled access and secret key pair that grants the authenticated user permission to the VAST bucket.

  To grant a user access to a bucket, and attach identity policies that define their permissions on that bucket, see [Managing S3 User Access from the VAST Web UI](https://kb.vastdata.com/documentation/docs/managing-s3-user-access-from-the-vast-web-ui-1) or [Managing S3 Access from the VAST CLI](https://kb.vastdata.com/documentation/docs/managing-s3-access-from-the-vast-cli) in the *VAST Cluster Administrator's Guide*.

  <Note>
    VAST recommends using identity policies and bucket policies to control S3 bucket access.

    * *Identity policies* are attached to users and groups, and defines the actions the users or groups can perform against buckets.
    * *Bucket policies* are attached to a view, and define actions that particular users or groups can perform against the view and files or directories under the view.

    For more information, see [Overview of S3 Access Management](https://kb.vastdata.com/documentation/docs/overview-of-s3-access-management) in the *VAST Cluster Administrator's Guide*.

    Identity and bucket policies support a subset of elements listed in [Amazon's IAM JSON Policy Reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html). For information about the required JSON format, supported elements and identity and bucket policy examples, see [Identity and Bucket Policy Reference](https://kb.vastdata.com/documentation/docs/identity-and-bucket-policy-reference-1) in the *VAST Cluster Administrator's Guide*.
  </Note>

  For source connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:ListBucket`
  * `s3:GetObject`

  For destination connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:PutObject`

VAST buckets are private by default. [Anonymous access](https://kb.vastdata.com/documentation/docs/overview-of-vast-cluster-s3-implementation-1#support-for-authenticated-or-anonymous-requests) can be explicitly enabled using bucket policies, but is not recommended.

## Create the connector

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **VAST**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Remote URL** (*required*): The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/`. For example: `vast://my-bucket/`

  If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/`. For example: `vast://my-bucket/my-folder/`

  <Note>
    For destination connectors, the relative directory structure of files is preserved when files are uploaded. Also, files are overwritten if they already exist at the destination.
  </Note>
* **Recursive** (source connector only): Check this box to access sub-folders within the bucket.
* **Max Objects** (source connector only): The maximum number of objects to index. This is useful for large buckets.
* **Endpoint URL** (*required*): The VAST storage endpoint URL. For example, `https://vast-cluster.my-company.com`.
* **Key** (*required*): The access key ID for the authenticated user.
* **Secret** (*required*): The secret access key corresponding to the preceding access key ID.


# Weaviate
Source: https://docs.unstructured.io/pipelines/destinations/weaviate



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Send processed data from Unstructured to Weaviate.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview): only [Weaviate Cloud](https://weaviate.io/developers/wcs) clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview): Weaviate Cloud clusters,
  [Weaviate installed locally](https://weaviate.io/developers/weaviate/quickstart/local),
  and [Embedded Weaviate](https://weaviate.io/developers/weaviate/installation/embedded) are supported.
* For Weaviate installed locally, you will need the name of the target collection on the local instance.
* For Embedded Weaviate, you will need the instance's connection URL and the name of the target collection on the instance.
* For Weaviate Cloud, you will need:

  <iframe title="YouTube video player" />

  * A Weaviate database instance. The following information assumes that you have a Weaviate Cloud (WCD) account with a Weaviate database cluster in that account.
    [Create a WCD account](https://weaviate.io/developers/wcs/quickstart#create-a-wcd-account). [Create a database cluster](https://weaviate.io/developers/wcs/quickstart#create-a-weaviate-cluster). For other database options, [learn more](https://weaviate.io/developers/weaviate/installation).
  * The URL and API key for the database cluster. [Get the URL and API key](https://weaviate.io/developers/wcs/quickstart#explore-the-details-panel).
  * The name of the target collection in the database. [Create a collection](https://weaviate.io/developers/wcs/tools/collections-tool).

    An existing collection is not required. At runtime, the collection behavior is as follows:

    For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview):

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your Weaviate cluster. If Unstructured generates embeddings,
      the new collection's name will be `U<short-workflow-id>_<short-embedding-model-name>_<number-of-dimensions>`.
      If Unstructured does not generate embeddings, the new collection's name will be `U<short-workflow-id`.

    For [Unstructured Ingest](/open-source/ingestion/overview):

    * If an existing collection name is specified, and Unstructured generates embeddings,
      but the number of dimensions that are generated does not match the existing collection's embedding settings, the run will fail.
      You must change your Unstructured embedding settings or your existing collection's embedding settings to match, and try the run again.
    * If a collection name is not specified, Unstructured creates a new collection in your Weaviate cluster. The new collection's name will be `Unstructuredautocreated`.

    If Unstructured creates a new collection and generates embeddings, you will not see an `embeddings` property in tools such as the Weaviate Cloud
    **Collections** user interface. To view the generated embeddings, you can run a Weaviate GraphQL query such as the following. In this query, replace `<collection-name>` with
    the name of the new collection, and replace `<property-name>` with the name of each additional available property that
    you want to return results for, such as `text`, `type`, `element_id`, `record_id`, and so on. The embeddings will be
    returned in the `vector` property.

    ```text theme={null}
    {
      Get {
        <collection-name> {
          _additional {
            vector
          }
          <property-name>
          <property-name>
        }
      }
    }
    ```

If [auto-schema](https://docs.weaviate.io/weaviate/config-refs/collections#auto-schema) is enabled in Weaviate (which it is by default),
Weaviate can infer missing properties and add them to the collection definition at run time. However, it is a Weaviate best practice to manually define as much
of the data schema in advance as possible, since manual definition gives you the most control.

The minimum viable schema for Unstructured includes only the `element_id` and `record_id` properties. The `text` and `type` properties should also be included, but they are technically optional.
If you are using Unstructured to generate embeddings, you must

The following code example shows how to use the [weaviate-client](https://pypi.org/project/weaviate-client/) Python package to create a
collection in a Weaviate Cloud database cluster with this minimum viable schema, and to specify that Unstructured will generate the embeddings for this collection.
To connect to a locally hosted Weaviate instance instead, call [weaviate.connect\_to\_local](https://docs.weaviate.io/weaviate/connections/connect-local).
To connect to Embedded Weaviate instead, call [weaviate.connect\_to\_embedded](https://docs.weaviate.io/weaviate/connections/connect-embedded).

```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
```

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

For objects in the `metadata` field that Unstructured produces and that you want to store in a Weaviate collection, be sure to follow
Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You could create corresponding properties in your collection's schema by using the following property names and data types:

```python theme={null}
import os
import weaviate
from weaviate.classes.init import Auth
import weaviate.classes.config as wvc

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.getenv("WEAVIATE_URL"),
    auth_credentials=Auth.api_key(api_key=os.getenv("WEAVIATE_API_KEY")),
)

collection = client.collections.create(
    name="MyCollection",
    properties=[
        wvc.Property(name="element_id", data_type=wvc.DataType.UUID),
        wvc.Property(name="record_id", data_type=wvc.DataType.TEXT),
        wvc.Property(name="text", data_type=wvc.DataType.TEXT),
        wvc.Property(name="type", data_type=wvc.DataType.TEXT),
        wvc.Property(
            name="metadata",
            data_type=wvc.DataType.OBJECT,
            nested_properties=[
                wvc.Property(name="is_extracted", data_type=wvc.DataType.TEXT),
                wvc.Property(
                    name="coordinates",
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(name="points", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="system", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="layout_width", data_type=wvc.DataType.NUMBER),
                        wvc.Property(name="layout_height", data_type=wvc.DataType.NUMBER),
                    ],
                ),
                wvc.Property(name="filetype", data_type=wvc.DataType.TEXT),
                wvc.Property(name="languages", data_type=wvc.DataType.TEXT_ARRAY),
                wvc.Property(name="page_number", data_type=wvc.DataType.TEXT),
                wvc.Property(name="image_mime_type", data_type=wvc.DataType.TEXT),
                wvc.Property(name="filename", data_type=wvc.DataType.TEXT),
                wvc.Property(
                    name="data_source",
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(name="url", data_type=wvc.DataType.TEXT),
                        wvc.Property(name="record_locator", data_type=wvc.DataType.TEXT),
                    ],
                ),
                wvc.Property(
                    name="entities", 
                    data_type=wvc.DataType.OBJECT,
                    nested_properties=[
                        wvc.Property(
                            name="items", 
                            data_type=wvc.DataType.OBJECT_ARRAY,
                            nested_properties=[
                                wvc.Property(name="entity", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="type", data_type=wvc.DataType.TEXT),
                            ],
                        ),
                        wvc.Property(
                            name="relationships", 
                            data_type=wvc.DataType.OBJECT_ARRAY,
                            nested_properties=[
                                wvc.Property(name="to", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="from", data_type=wvc.DataType.TEXT),
                                wvc.Property(name="relationship", data_type=wvc.DataType.TEXT),
                            ],
                        ),
                    ],
                ),
            ],
        ),
    ],
    vectorizer_config=None, # Unstructured will generate the embeddings instead of Weaviate.
)

client.close()
```

Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

See also:

* [Collection schema](https://weaviate.io/developers/weaviate/config-refs/schema)
* [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

To create the destination connector:

1. On the sidebar, click **Connectors**.
2. Click **Destinations**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Weaviate**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for the connector.
* **Cluster URL** (*required*): The URL of the Weaviate database cluster.
* **Collection Name**: The name of the target collection within the cluster. If no value is provided, see the beginning of this article
  for the behavior at run time.
* **API Key** (*required*): The API key provided by Weaviate to access the cluster.


# Jobs
Source: https://docs.unstructured.io/pipelines/jobs



## Jobs dashboard

<img alt="Jobs in the sidebar" />

To view the jobs dashboard, on the sidebar, click **Jobs**.

The jobs dashboard provides a centralized view for managing and monitoring the execution of data processing tasks within your workflows.

The jobs dashboard lists each job and its associated **Status**, **Job ID**, **Created** date and time, **Workflow** name, and **Runtime** duration.

Each job's status, shown in the **Status** column, can be:

<img alt="Pending job icon" /> **Pending**: The job's data is currently not attempting to be processed.
<img alt="In-progress job icon" /> **In Progress**: The job's data is attempting to be processed.
<img alt="Fully finished job icon" /> **Finished**: 100% of the job's data has been successfully processed.
<img alt="Partially finished job icon" /> **Finished**: 90% to 99% of the job's data has been sucessfully processed.
<img alt="Failed job icon" /> **Failed**: Less than 90% of the job's data has been successfully processed.

## Run a job

<Warning>
  You must first have an existing workflow to run a job against.

  If you do not have an existing workflow, stop. [Create a workflow](/pipelines/workflows#create-a-workflow), and then return here.

  To see your existing workflows, on the sidebar, click **Workflows**.
</Warning>

To run a job, on the sidebar, click **Workflows**, and then click **Run** in the row for the workflow that you want to run.

## Monitor a job

<img alt="Completed job" />

The job details pane is a comprehensive section for monitoring the specific details of jobs executed within a particular workflow. To access this pane, click the specific job on the jobs dashboard.

Clicking the **Details** button shows:

* The job's ID.
* The job's start date.
* Information about what happened in each of the job's stages.

Clicking the **Errors** button provides any available information about what might have gone wrong with the job or in any of its stages.


# Overview
Source: https://docs.unstructured.io/pipelines/overview



<Tip>To start using Unstructured Pipelines right away, skip ahead to the [quickstarts](/pipelines/quickstart) now!</Tip>

## What is Unstructured Pipelines?

Unstructured Pipelines is a no-code user interface, pay-as-you-go platform for transforming your unstructured data into data that is ready for retrieval-augmented generation (RAG).

<Frame>
  <img alt="Unstructured Pipelines example" />
</Frame>

This 90-second video provides a brief overview of Unstructured Pipelines:

<iframe title="YouTube video player" />

<Icon icon="blog" />  [Read the announcement](https://unstructured.io/blog/introducing-unstructured-platform-the-enterprise-etl-platform-for-the-genai-tech-stackintroducing-unstructured-platform-beta-the-enterprise-etl-platform-for-the-genai-tech-stack).

## How does it work?

To get your data RAG-ready, Unstructured moves it through the following process:

```mermaid theme={null}
flowchart LR
    Connect-->Route-->Transform-->Chunk-->Enrich-->Embed-->Persist
```

<Steps>
  <Step title="Connect">
    Unstructured offers multiple [source connectors](/pipelines/sources/overview) to connect to your data in its existing location.
  </Step>

  <Step title="Route">
    Routing determines which strategy Unstructured uses to transform your documents into Unstructured's canonical JSON schema. Unstructured provides four [partitioning](/concepts/partitioning) strategies for document transformation, as follows.

    Unstructured recommends that you choose the **Auto** partitioning strategy in most cases. With **Auto**, Unstructured does all
    the heavy lifting, optimizing at runtime for the highest quality at the lowest cost page-by-page.

    You should consider the following additional strategies only if you are absolutely sure that your documents are of the same
    type. Each of the following strategies are best suited for specific situations. Choosing one of these
    strategies other than **Auto** for sets of documents of different types could produce undesirable results,
    including reduction in transformation quality.

    * **VLM**: For the highest-quality transformation of these file types: `.bmp`, `.gif`, `.heic`, `.jpeg`, `.jpg`, `.pdf`, `.png`, `.tiff`, and `.webp`.
    * **High Res**: For all other [supported file types](/pipelines/supported-file-types), and for the generation of bounding box coordinates.
    * **Fast**: For text-only documents.

    The **Auto** partitioning strategy routes each file as a complete unit to the appropriate partitioning strategy (**VLM**, **High Res**, or **Fast**)
    based on the preceding file types. Additionally, for `.pdf` files, the **Auto** partitioning strategy routes these files' pages
    on a page-by-page basis, as follows:

    * A page is routed to **Fast** when it contains only embedded text and no images or tables are detected.
    * All other kinds of pages are routed to **VLM** or **High Res**, depending on the complexity of a page's
      content. Unstructured constantly optimizes its proprietary algorithm for routing to **VLM** or **High Res** in these cases.
  </Step>

  <Step title="Transform">
    Your source document is transformed into Unstructured's canonical JSON schema. Regardless of the input document, this JSON schema gives you a [standardized output](/concepts/document-elements). It contains more than 20 elements, such as `Header`, `Footer`, `Title`, `NarrativeText`, `Table`, `Image`, and many more. Each document is wrapped in extensive metadata so you can understand languages, file types, sources, hierarchies, and much more.
  </Step>

  <Step title="Chunk">
    Unstructured provides these [chunking](/concepts/chunking) strategies:

    * **Basic** combines sequential elements up to specified size limits. Oversized elements are split, while tables are isolated and divided if necessary. Overlap between chunks is optional.
    * **By Title** uses semantic chunking, understands the layout of the document, and makes intelligent splits.
    * **By Page** attempts to preserve page boundaries when determining the chunks' contents.
    * **By Similarity** uses an embedding model to identify topically similar sequential elements and combines them into chunks.
  </Step>

  <Step title="Enrich">
    Images and tables can be optionally summarized. This generates enriched content around the images or tables that were parsed during the transformation process.
  </Step>

  <Step title="Embed">
    Unstructured uses optional third-party [embedding](/concepts/embedding) providers such as OpenAI.
  </Step>

  <Step title="Persist">
    Unstructured offers multiple [destination connectors](/pipelines/destinations/overview), including all major vector databases.
  </Step>
</Steps>

To simplify this process and provide it as a no-code solution, Unstructured brings together these key concepts:

```mermaid theme={null}
flowchart LR
    
    SC[Source Connectors]
    DC[Destination Connectors]
    W[Workflow]
    J[Jobs]
    
    SC --> W
    DC --> W
    W --> J
```

<Steps>
  <Step title="Source Connectors">
    [Source connectors](/pipelines/sources/overview) to ingest your data into Unstructured for transformation.
  </Step>

  <Step title="Destination Connectors">
    [Destination connectors](/pipelines/destinations/overview) tell Unstructured where to write your transformed data to.
  </Step>

  <Step title="Workflow">
    A [workflow](/pipelines/workflows) connects sources to destinations and provide chunking, embedding, and scheduling options.
  </Step>

  <Step title="Jobs">
    [Jobs](/pipelines/jobs) enable you to monitor data transformation progress.
  </Step>
</Steps>

## What support is there for compliance?

The platform is designed to comply with SOC 2 Type 1, SOC 2 Type 2, HIPAA, GDPR, ISO 27001, FedRAMP, and CMMC 2.0 Level 2. It has support for over 50 languages.
For details, see the [Unstructured Trust Portal](https://trust.unstructured.io/).

## How am I billed?

To use Unstructured Pipelines, you must have an Unstructured account for billing purposes.

Unstructured offers different account types with different pricing plans:

* <Icon icon="person" />  **Let's Go** and **Pay-As-You-Go** - A single user, with a single workspace, hosted alongside other accounts on Unstructured's cloud infrastructure.
* <Icon icon="building" />  **Business** - Multiple users and workspaces, with three options:

  * <Icon icon="people-group" />  **Business SaaS** - Hosted alongside other accounts on Unstructured's cloud infrastructure.
  * <Icon icon="shield-halved" />  **Dedicated instance** - Hosted within a virtual private cloud (VPC) running inside Unstructured's cloud infrastructure. Dedicated instances are isolated from all other accounts, for additional security and control.
  * <Icon icon="shield" />  **In-VPC** - Hosted within your own VPC on your own cloud infrastructure.

  **Business** accounts also allow for robust customization of Unstructured's features for your unique needs.

For more details, see the [Unstructured Pricing](https://unstructured.io/pricing) page.

To upgrade your account from **Let's Go** or **Pay-As-You-Go** to **Business**,
email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

Some of these plans have billing details that are determined on a per-page basis.

Unstructured calculates a page as follows:

* For these file types, a page is a page, slide, or image: `.pdf`, `.pptx`, and `.tiff`.
* For `.docx` files that have page metadata, Unstructured calculates the number of pages based on that metadata.
* For all other file types, Unstructured calculates the number of pages as the file's size divided by 100 KB.
* For non-file data, Unstructured calculates a page as 100 KB of incoming data to be processed.

## How do I get started?

Skip ahead to the [quickstart](/pipelines/quickstart).

## Questions? Need help?

* For general questions about Unstructured products and pricing, email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).
* For technical support for Unstructured accounts, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# Unstructured Pipelines quickstarts
Source: https://docs.unstructured.io/pipelines/quickstart



## Local file quickstart

This quickstart shows how, in just a few minutes, you can use Unstructured Pipelines to quickly and easily see Unstructured's
best-in-class transformation results for a single file that is stored on your local computer.

<Tip>
  This quickstart focuses on a single, local file for ease-of-use demonstration purposes.

  To use Unstructured later to do
  large-scale batch processing of multiple files and semi-structured data that are stored in remote locations,
  [skip over](/pipelines/quickstart#remote-quickstart) to the remote quickstart after you finish this one.
</Tip>

If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

<Note>
  If you already have an Unstructured **Pay-As-You-Go** or **Business SaaS** account, you are already signed up for Unstructured.
  Sign in to your existing Unstructured **Pay-As-You-Go** or **Business SaaS** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

  If you already have an Unstructured **dedicated instance** or **in-VPC** deployment, your sign-in link will be unique to your deployment.
  If you're not sure what your unique sign-in link is, see your Unstructured account administrator, or email Unstructured Support at
  [support@unstructured.io](mailto:support@unstructured.io).
</Note>

Do the following:

1. After you are signed in, the **Start** page appears.

2. In the **Welcome** area, do one of the following:

   * Click one of the sample files, such as **realestate.pdf**, to have Unstructured parse and transform that sample file.
   * Click **Browse files**, or drag and drop a file onto **Drop file to test**, to have Unstructured parse and transform your own file.

     If you choose to use your own file, the file must be 50 MB or less in size. Also, the file must be one of the following supported file types:

     | File extension |
     | -------------- |
     | `.bmp`         |
     | `.csv`         |
     | `.doc`         |
     | `.docx`        |
     | `.eml`         |
     | `.epub`        |
     | `.heic`        |
     | `.html`        |
     | `.jpeg`        |
     | `.jpg`         |
     | `.md`          |
     | `.msg`         |
     | `.odt`         |
     | `.org`         |
     | `.p7s`         |
     | `.pdf`         |
     | `.png`         |
     | `.ppt`         |
     | `.pptx`        |
     | `.rst`         |
     | `.rtf`         |
     | `.tif`         |
     | `.tiff`        |
     | `.tsv`         |
     | `.txt`         |
     | `.xls`         |
     | `.xlsx`        |
     | `.xml`         |

   <img alt="Welcome interface on the Start page" />

3. After Unstructured has finished parsing and transforming the file (a process known as
   [partitioning](/concepts/partitioning)), you will see the file's contents in the
   **Preview** pane in the center and Unstructured's results in the **Result** pane on the right.

   <img alt="Unstructured's parse and transform results" />

4. The **Result** pane shows a formatted view of Unstructured's results by default. This formatted view is designed for human
   readability. To see the underlying JSON view of the results, which is designed for RAG and agentic AI,
   click **JSON** at the top of the **Result** pane.
   [Learn about what's in the JSON view](/concepts/document-elements).

   <img alt="Switching to the JSON view of the results" />

5. Unstructured's initial results are based on its **High Res** [partitioning strategy](/concepts/partitioning), which
   begins processing the file's contents and converting these contents into a series of Unstructured
   [document elements and metadata](/concepts/document-elements). This partitioning strategy provides good results overall, depending on the complexity of the file's contents.
   This partitioning strategy also generates a bounding box for each detected object in the file. A *bounding box* is
   an imaginary rectangular box drawn around the object to show its location and extent within the file.

   After the High Res partitioning results are shown, Unstructured begins improving these initial results by
   using vision language models (VLMs) to apply a series of generative refinements known as *enrichments*. These
   enrichments include:

   * An [image description](ui/enriching/image-descriptions) enrichment, which uses a VLM to provide a text-based summary of the contents of each detected image.
   * A [generative OCR](/concepts/enriching/generative-ocr) enrichment, which uses a VLM to improve the accuracy of each block of initially-processed text.
   * A [table to HTML](/concepts/enriching/table-to-html) enrichment, which uses a VLM to provide an HTML-structured representation of each detected table.

   While these enrichments are being applied, a banner appears at the top of the **Result** pane.

   <img alt="Updating the initial results with enrichments" />

   To see these enrichments applied to the initial results, click **Update results** in the banner as soon as this button appears,
   which might take up to a minute or more.

   <img alt="Seeing the initial results updated with the enrichments" />

   <Warning>
     Each page that Unstructured processes by using this approach is counted as two pages for usage and billing purposes.

     This is because Unstructured processes each page once with its **High Res** partitioning strategy and then reprocesses each
     page with a VLM to improve the quality, accuracy, and relevance of the initial partitioning results.
     The final results of these two processing passes for each page count as two pages for usage and billing purposes.
     This two-pass process happens regardless of whether you click **Update results** in the banner.

     This two-page usage and billing behavior is a known issue and will be addressed in a future release.
   </Warning>

6. To synchronize the scrolling of the **Preview** pane's selected contents with the **Result** pane's **Formatted** results,
   rest your mouse pointer anywhere inside the contents of the **Preview** pane until a bounding box appears.
   Then click the bounding box. Unstructured automatically scrolls the **Result** pane's **Formatted**
   results to match the selected bounding box. (You cannot synchronize the scrolling of the **JSON** results.)

   <img alt="Selecting a bounding box" />

   To show all of the bounding boxes in the **Preview** pane at once, turn on the **Show all bounding boxes** toggle at the top of the **Preview** pane.
   You can now click any of the bounding boxes without first needing to rest your mouse pointer on them to show them.

   <img alt="Showing all bounding boxes" />

You can also do the following:

* To download the JSON view of the results as a local JSON file, click the download icon to the left of the **Formatted** and **JSON** buttons in the **Result** pane.
  (You cannot download the formatted view of the results.)

  <img alt="Downloading the results as a local JSON file" />

* To have Unstructured partition a different file, click **Add new file** in the **Files** pane on the left, and then browse to and select the target file.

* To view the results for a file that was previously partitioned during this session, click the file's name in the **Recent files** list in the **Files** pane.

* To return to the **Start** page, click the **X** (close) button at the left on the title bar, next to **Transform**.

* To have Unstructured do more—such as
  [chunking](/concepts/chunking), [embedding](/concepts/embedding),
  applying additional kinds of [enrichments](/concepts/enriching/overview), and
  processing larger files and semi-structured data in batches at scale—click
  **Edit in Workflow Editor** at the right on the title bar, and then [skip over to the walkthrough](/pipelines/walkthrough-2).

  <img alt="Switching to the workflow editor" />

What's next?

* <Icon icon="code" />  [Learn how to extract structured data in a custom format from your local file](/concepts/structured-data-extractor/using-the-structured-data-extractor#use-the-structured-data-extractor-from-the-start-page).
* <Icon icon="plus" />  [Learn how to add chunking, embeddings, custom structured data extraction, and additional enrichments to your local file results](/pipelines/walkthrough-2).
* <Icon icon="database" />  [Learn how to do large-scale batch processing of multiple files and semi-structured data that are stored in remote locations instead](/pipelines/quickstart#remote-quickstart).
* <Icon icon="desktop" />  [Learn more about Unstructured Pipelines](/pipelines/overview).

***

## Remote quickstart

The following quickstart shows you how to use Unstructured Pipelines to process remote files (or data).

The requirements are as follows.

* A compatible source (input) location that contains your data for Unstructured to process. [See the list of supported source types](/pipelines/connectors#sources).

  If your source (input) location is not in this list, or if you do not yet have any source locations for Unstructured to process, **stop here** and
  skip over to the [Dropbox source connector quickstart](/pipelines/sources/dropbox-source-quickstart) instead. This quickstart
  guides you through the process of creating a free Dropbox account, uploading your files to Dropbox,
  and creating a source connector to connect Unstructured to those files.

* For document-based source locations, compatible files in that location. [See the list of supported file types](/pipelines/supported-file-types). If you do not have any files available, you can download some from the [example-docs](https://github.com/Unstructured-IO/unstructured-ingest/tree/main/example-docs) folder in the Unstructured repo on GitHub.

* A compatible destination (output) location for Unstructured to put the processed data. [See the list of supported destination types](/pipelines/connectors#destinations).

  If your destination (output) location is not in this list, or if you do not yet have any destination locations for Unstructured to send its processed data, **stop here** and
  skip over to the [Pinecone destination connector quickstart](/pipelines/destinations/pinecone-destination-quickstart) instead. This quickstart
  guides you through the process of creating a free Pinecone account
  and creating a destination connector to connect Unstructured to a Pinecone dense serverless index within your Pinecone account.

<iframe title="YouTube video player" />

<Steps>
  <Step title="Sign up and sign in">
    1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
       After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

       <Note>
         To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/pipelines/overview#how-am-i-billed%3F).
       </Note>

    2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

       <Note>
         For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
         or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
       </Note>
  </Step>

  <Step title="Set the source (input) location">
    <img alt="Sources in the sidebar" />

    1. From your Unstructured  dashboard, in the sidebar, click **Connectors**.
    2. Click **Sources**.
    3. Click **New** or **Create Connector**.
    4. For **Name**, enter some unique name for this connector.
    5. In the **Provider** area, click the source location type that matches yours.
    6. Click **Continue**.
    7. Fill in the fields with the appropriate settings. [Learn more](/pipelines/sources/overview).
    8. If a **Continue** button appears, click it, and fill in any additional settings fields.
    9. Click **Save and Test**.
  </Step>

  <Step title="Set the destination (output) location">
    <img alt="Destinations in the sidebar" />

    1. In the sidebar, click **Connectors**.
    2. Click **Destinations**.
    3. Click **New** or **Create Connector**.
    4. For **Name**, enter some unique name for this connector.
    5. In the **Provider** area, click the destination location type that matches yours.
    6. Click **Continue**.
    7. Fill in the fields with the appropriate settings. [Learn more](/pipelines/sources/overview).
    8. If a **Continue** button appears, click it, and fill in any additional settings fields.
    9. Click **Save and Test**.
  </Step>

  <Step title="Define the workflow">
    <img alt="Workflows in the sidebar" />

    1. In the sidebar, click **Workflows**.

    2. Click **New Workflow**.

    3. Next to **Build it for Me**, click **Create Workflow**.

       <Note>If a radio button appears instead of **Build it for Me**, select it, and then click **Continue**.</Note>

    4. For **Workflow Name**, enter some unique name for this workflow.

    5. In the **Sources** dropdown list, select your source location from Step 3.

    6. In the **Destinations** dropdown list, select your destination location from Step 4.

       <Note>You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.</Note>

    7. Click **Continue**.

    8. The **Reprocess All** box applies only to blob storage connectors such as the Amazon S3, Azure Blob Storage, and Google Cloud Storage connectors:

       * Checking this box reprocesses all documents in the source location on every workflow run.
       * Unchecking this box causes only new documents that are added to the source location, or existing documents that are updated in the source location (as determined by checking whether the file's version has changed), since the last workflow run to be processed on future runs. Previously processed documents are not processed again. However:

         * Even if this box is unchecked, a renamed file is always treated as a new file, regardless of whether the file's original contents have changed.
         * Even if this box is unchecked, a file that is removed but is added back later with the same file name is processed on future runs only if the file's contents have changed since the file was originally processed.

    9. Click **Continue**.

    10. If you want this workflow to run on a schedule, in the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select **Don't repeat**.

    11. Click **Complete**.
  </Step>

  <Step title="Process the documents">
    <img alt="Workflows in the sidebar" />

    1. If you did not choose to run this workflow on a schedule in Step 5, you can run the workflow now: on the sidebar, click **Workflows**.
    2. Next to your workflow from Step 5, click **Run**.
  </Step>

  <Step title="Monitor the processing job">
    <img alt="Select a job" />

    <img alt="Completed job" />

    1. In the sidebar, click **Jobs**.
    2. In the list of jobs, wait for the job's **Status** to change to **Finished**.
    3. Click the row for the job.
    4. After **Overview** displays **Finished**, go to the next Step.
  </Step>

  <Step title="View the processed data">
    Go to your destination location to view the processed data.
  </Step>
</Steps>

Learn more about Unstructured [source connectors](/pipelines/sources/overview),
[destination connectors](/pipelines/destinations/overview),
[workflows](/pipelines/workflows),
[jobs](/pipelines/jobs), and
[managing your account](/pipelines/account/overview).


# Azure Blob Storage
Source: https://docs.unstructured.io/pipelines/sources/azure-blob-storage



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Azure Blob Storage.

The following video shows how to fulfill the minimum set of Azure Storage account requirements:

<iframe title="YouTube video player" />

<Note>
  If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

  * **Read** and **List** for reading from the container only.
  * **Write** and **List** for writing to the container only.
  * **Read**, **Write**, and **List** for both reading from and writing to the container.
</Note>

Here are some more details about these requirements:

* An Azure account. To create one, [learn how](https://azure.microsoft.com/pricing/purchase-options/azure-account).

  <iframe title="YouTube video player" />

* An Azure Storage account, and a container within that account. [Create a storage account](https://learn.microsoft.com/azure/storage/common/storage-account-create). [Create a container](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal).

  <iframe title="YouTube video player" />

  <iframe title="YouTube video player" />

* The Azure Storage remote URL, using the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* An SAS token (recommended), access key, or connection string for the Azure Storage account.  [Create an SAS token (recommended)](https://learn.microsoft.com/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens). [Get an access key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage#view-account-access-keys). [Get a connection string](https://learn.microsoft.com/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-an-azure-storage-account).

  Create an SAS token (recommended):

  <iframe title="YouTube video player" />

  Get an access key or connection string:

  <iframe title="YouTube video player" />

<h2>
  Set up Enterprise Connect authentication
</h2>

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see [Enterprise Connect for Azure](/business/azure/enterprise-connect).

To configure an Azure Blob Storage connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

1. Create an App Registration for Unstructured in Microsoft Entra ID.

   In your Azure subscription, follow the instructions in [How to register an app in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, `unstructured-connector`). For **Supported account types**, select **Single tenant only**.

   You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the [single-tenant scenario](https://learn.microsoft.com/en-us/entra/identity-platform/single-and-multi-tenant-apps) as defined by Microsoft.

2. Add a federated identity credential to the App Registration.

   Follow the instructions in [Configure an app to trust an external identity provider](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-create-trust) in the Microsoft Entra documentation. Navigate to your App Registration, select **Certificates & secrets** in the left navigation pane, select the **Federated credentials** tab, and select **Add credential**.

   For **Federated credential scenario**, select **Other issuer**.

   Set the following values:

   | Field        | Value                                                                                                                                                                                                                                                                                                            |
   | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Name**     | A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.                                                                                                                                                                                     |
   | **Issuer**   | The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)                                                                                                                                                       |
   | **Subject**  | The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error. |
   | **Audience** | Set this to `api://AzureADTokenExchange`.                                                                                                                                                                                                                                                                        |

   Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

3. Add a role assignment to grant your App Registration access to your Azure Blob Storage account.

   See [Assign Azure roles using the Azure portal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) in the Azure documentation. Use the following values:

   * **Scope**: the Azure Blob Storage account that contains the data you want the connector to access.
   * **Role**: select **Storage Blob Data Reader** for a source, or **Storage Blob Data Contributor** for a destination.
   * **Members**: select **User, group, or service principal**, then search for and select the App Registration you created in Step 1.

   When you reach the **Review + assign** tab, click **Review + assign** to complete the assignment.

4. Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the **Overview** page of your App Registration in the [Microsoft Entra admin center](https://entra.microsoft.com).

   * The **Tenant ID** (also called Directory ID) for your Azure subscription.
   * The **Client ID** of your App Registration.

Next, create the connector by following the steps in the **Create the  connector** section on this page.

## Create the source connector

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Click **New** or **Create Connector**.
4. For **Name**, enter a unique name for this connector.
5. In the **Provider** area, click **Azure Blob Storage**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Recursive** (source connector only): Check this box to recursively access files from subfolders within the container.

* **Remote URL** (*required*): The Azure Storage remote URL, with the format `az://<container-name>/<path/to/file/or/folder/in/container/as/needed>`

  For example, if your container is named `my-container`, and there is a folder in the container named `my-folder`, the
  Azure Storage remote URL would be `az://my-container/my-folder/`.

* **Account Name**: For SAS token authentication, account key authentication, and Enterprise Connect (Workload Identity Federation), the Azure Storage account name.

* **SAS Token**: For SAS token authentication, the SAS token for the Azure Storage account.

* **Account Key**: For account key authentication, the key for the Azure Storage account.

* **Connection String**: For connection string authentication, the connection string for the Azure Storage account.

If you're using **Enterprise Connect (Workload Identity Federation)** ([dedicated instance](/business/dedicated-instances/overview) only), fill in the following fields:

* For **Authentication Method**, select **Enterprise Connect (Workload Identity Federation)**.
* For **Tenant ID**, enter the Tenant ID (also called Directory ID) for your Azure subscription.
* For **Client ID**, enter the Client ID of your App Registration.


# Box
Source: https://docs.unstructured.io/pipelines/sources/box

Ingest your files into Unstructured from Box.

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

## Requirements

You will need:

<iframe title="YouTube video player" />

1. Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).

2. A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).

3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

   * In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
   * In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
   * Click **Save Changes** before continuing.

4. Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).

5. Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:

   * In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com`).
   * **Share** your Box account's target folder with the copied service account's email address as a **Co-owner** or **Editor**.
   * Note the remote URL to the target folder, which takes the format `box://<path/to/folder/in/account>`.

6. The private key configuration JSON file for the Box Custom App, or a string that contains this file's contents.

   * To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location.
   * To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as
     a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself).
     To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
     In this command, replace `<path-to-downloaded-key-file>` with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.

     * For macOS or Linux:

       ```text theme={null}
       tr -d '\n' < <path-to-downloaded-key-file>
       ```

     * For Windows:

       ```text theme={null}
       (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
       ```

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

Permissions are defined in Box through *collaborations*, which function similar to access control lists. Collaboration objects are attached to folders, and those collaborations cascade down to everything inside that folder. A file's effective permissions come from the chain of ancestor folders above it. For more information, see [Collaborations overview](https://developer.box.com/guides/collaborations/) in the Box developer documentation.

To determine a file's effective permissions, Unstructured traverses the full folder hierarchy for each file, collecting collaborations at every level, and merging them. It also includes collaborations applied directly to the file itself. For more information, see [List folder collaborations](https://developer.box.com/reference/get-folders-id-collaborations/)  and [List file collaborations](https://developer.box.com/reference/get-files-id-collaborations/) in the Box developer documentation.

When Unstructured compiles a file's effective permissions:

* If the same user or group appears at multiple levels with different roles, Unstructured chooses the least-restrictive role.
* Unstructured does not include the following in the permission metadata. No sentinel value is written into the permission metadata.
  * `all_users_group`: Box's built-in group representing every user in the enterprise.
  * `is_access_only`: Collaboration property that determines whether, for collaborators, to display the items in the All Files list and let them see the path to the root folder for the shared item.
* Unstructured only includes collaborations with a status of `accepted`. It does not include collaborations with a status of `pending` or `rejected`.

Unstructured includes a maximum of 500 permissions per file.

Unstructured maps Box roles to `read`, `update`, and `delete` access:

| Box role             | Box access                                               | Unstructured permission metadata |
| -------------------- | -------------------------------------------------------- | -------------------------------- |
| `owner`              | Full access                                              | `read`, `update`, `delete`       |
| `co-owner`           | Full access except transfer ownership                    | `read`, `update`, `delete`       |
| `editor`             | View, download, upload, edit, delete, copy, move, rename | `read`, `update`                 |
| `viewer`             | Preview, download, comment                               | `read`                           |
| `previewer`          | Preview                                                  | `read`                           |
| `viewer uploader`    | Preview, download, upload                                | `read`                           |
| `previewer uploader` | Preview, upload                                          | `read`                           |
| `uploader`           | Upload and see item names                                | excluded entirely                |

Note the following:

* Unstructured excludes the `uploader` role because it confers no `read`, `update`, or `delete` access.
* Unstructured maps the Box `editor` role to `read` and `update` access but not `delete`, even though that role does allow deleting files.

For more information, see [Understanding collaborator permission level](https://support.box.com/hc/en-us/articles/360044196413-Understanding-Collaborator-Permission-Levels) at Box Support.

### Identifier format

Users and groups are identified by the numeric ID that Box assigns them. To retrieve information about a specific user or group, use appropriate Box API with the corresponding ID:

* [Get user](https://developer.box.com/reference/get-users-id/)
* [Get group](https://developer.box.com/reference/get-groups-id/)

### Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["11446688", "11336699"],
                            "groups": ["11223344"]
                        }
                    },
                    {
                        "update": {
                            "users": ["11446688"],
                            "groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

## Creating the connector

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Box**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Data URL** (*required*): The URL to the target Box folder. This URL must take the format `box://<path/to/folder/in/account>`.
* **Recursive**: Check this box to recursively access files from subfolders within the taget Box folder. This box is unchecked by default.
* **Config** (*required*): The contents of the private key configuration JSON file for the Box Custom App with access to the target Box folder. This file's contents must be provided as a single string.

## Learn more

* <Icon icon="blog" />  [Finding Needles in a Haystack: PII Detection at Scale with Unstructured, Box, and Elasticsearch](https://unstructured.io/blog/finding-needles-in-a-haystack-pii-detection-at-scale-with-unstructured-box-and-elasticsearch)


# Confluence
Source: https://docs.unstructured.io/pipelines/sources/confluence



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Confluence.

The requirements are as follows.

* A [Confluence Cloud account](https://www.atlassian.com/software/confluence/pricing) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html).

* The site URL for your [Confluence Cloud account](https://community.atlassian.com/t5/Confluence-questions/confluence-cloud-url/qaq-p/1157148) or
  [Confluence Data Center installation](https://confluence.atlassian.com/confkb/how-to-find-your-site-url-to-set-up-the-confluence-data-center-and-server-mobile-app-938025792.html).

* A user in your [Confluence Cloud account](https://confluence.atlassian.com/cloud/invite-edit-and-remove-users-744721624.html) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/add-and-invite-users-138313.html).

* The user must have the correct permissions in your
  [Conflunce Cloud account](https://support.atlassian.com/confluence-cloud/docs/what-are-confluence-cloud-permissions-and-restrictions/) or
  [Confluence Data Center installation](https://confluence.atlassian.com/doc/permissions-and-restrictions-139557.html) to
  access the target spaces and pages.

* One of the following:

  * For Confluence Cloud or Confluence Data Center, the target user's name or email address, and password.
    [Change a Confluence Cloud user's password](https://support.atlassian.com/confluence-cloud/docs/change-your-confluence-password/).
    [Change a Confluence Data Center user's password](https://confluence.atlassian.com/doc/change-your-password-139416.html).
  * For Confluence Cloud only, the target user's name or email address, and API token.
    [Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
  * For Confluence Data Center only, the target user's personal access token (PAT).
    [Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).

* Optionally, the keys (not display names) of the specific [spaces](https://support.atlassian.com/confluence-cloud/docs/navigate-spaces/) in the Confluence instance to access. To get a space's key,
  which is different from a space's display name, open the space in your web browser and look at the URL. The space's key is the part of the URL after `spaces/` but before the next `/` after that.

The following video provides related setup information for Confluence Cloud:

<iframe title="YouTube video player" />

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "11111:11111111-1111-1111-1111-111111111111"
                            ],
                            "groups": [
                                "22222222-2222-2222-2222-22222222",
                                "33333333-3333-3333-3333-33333333"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "44444:44444444-4444-4444-4444-44444444",
                                "55555:55555555-5555-5555-5555-55555555"
                            ],
                            "groups": [
                                "66666666-6666-6666-6666-66666666",
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "77777:77777777-7777-7777-7777-77777777"
                            ],
                            "groups": [
                                "88888888-8888-8888-8888-88888888"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Confluence user, use the user's ID (also known as their *account ID*) from the preceding output to call the
[GET /wiki/rest/api/user](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-users/#api-wiki-rest-api-user-get)
operation in the Confluence REST API.

To look up information about a particular Confluence group, use the group's ID from the preceding output to call the
[GET /wiki/rest/api/group/by-id](https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-group/#api-wiki-rest-api-group-by-id-get)
operation in the Confluence REST API.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Confluence**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **URL** (*required*): The target Confluence site's URL.
* For username and password authentication: for **Authentication Method**, select **Username and Password**. Then enter the username into the **Username** field and the password into the **Password** field.
* For API token authentication: for **Authentication Method**, select **Username and API Token**. Then enter the username into the **Username** field and the API token into the **API Token** field.
* For personal access token (PAT) authentication: for **Authentication Method**, select **Personal Access Token**. Then enter the PAT into the **Personal Access Token** field.
* **Cloud**: Check this box if you are using Confluence Cloud. By default this box is unchecked.
* **Max number of spaces**: The maximum number of Confluence spaces to access within the Confluence Cloud instance.
  The default is 500 unless otherwise specified.
* **Max number of docs per space**: The maximum number of documents to access within each space.
  The default is 150 unless otherwise specified.
* **List of spaces**: A comma-separated string that lists the keys (not display names) of all of the spaces to access, for example: `luke,paul`.
  By default, if no space keys are specified, and the **Max Number of Spaces** is reached for the instance, be aware that you might get
  unexpected results.
* **Extract inline images**: Check this box to download images and replace the HTML content with Base64-encoded images. By default, this box is unchecked.
* **Extract files**: Check this box to download any embedded files in pages. By default, this box is unchecked.


# Couchbase
Source: https://docs.unstructured.io/pipelines/sources/couchbase



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Couchbase.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Couchbase Capella clusters are supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), Couchbase Capella clusters and local Couchbase server deployments are supported.

<iframe title="YouTube video player" />

For Couchbase Capella, you will need:

* A [Couchbase Capella account](https://docs.couchbase.com/cloud/get-started/create-account.html#sign-up-free-tier).
* A [Couchbase Capella cluster](https://docs.couchbase.com/cloud/get-started/create-account.html#getting-started).
* A [bucket](https://docs.couchbase.com/cloud/clusters/data-service/manage-buckets.html#add-bucket),
  [scope](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-scope),
  and [collection](https://docs.couchbase.com/cloud/clusters/data-service/scopes-collections.html#create-collection)
  on the cluster.
* The cluster's [public connection string](https://docs.couchbase.com/cloud/get-started/connect.html#connect-from-sdk-cbsh-cli-or-ide).
* The [cluster access name (username) and secret (password)](https://docs.couchbase.com/cloud/clusters/manage-database-users.html#create-database-credentials).
* [Incoming IP address allowance](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) for the cluster.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

For a local Couchbase server, you will need:

* [Installation of a local Couchbase server](https://docs.couchbase.com/server/current/getting-started/start-here.html).
* [Connection details](https://docs.couchbase.com/server/current/guides/connect.html) to the local Couchbase server.

To learn more about how to set up a Couchbase cluster and play with data, refer to this [tutorial](https://developer.couchbase.com/tutorial-quickstart-flask-python).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Couchbase**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bucket** (*required*): The name of the bucket to connect to on the cluster.
* **Connection string** (*required*): The connection string for the Couchbase cluster.
* **Scope**: The name of the scope in the bucket. The default is `_default` if not otherwise specified.
* **Collection**: The name of the collection in the scope. The default is `_default` if not otherwise specified.
* **Batch Size**: The maximum number of documents per transmitted batch. The default is `50` if not otherwise specified.
* **Collection ID key** (source connector only): The name of the collection field that contains the document ID. The default is `id` if not otherwise specified
* **Username** (*required*): The cluster access name (username) that has access to the cluster.
* **Password** (*required*): The password for the corresponding username.

## Learn more

* <Icon icon="blog" />  [Couchbase Integration in Unstructured Platform](https://unstructured.io/blog/couchbase-integration-in-unstructured-platform)


# Databricks Volumes
Source: https://docs.unstructured.io/pipelines/sources/databricks-volumes



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Databricks Volumes.

The requirements are as follows.

* A Databricks account on [AWS](https://docs.databricks.com/getting-started/free-trial.html),
  [Azure](https://learn.microsoft.com/azure/databricks/getting-started/), or
  [GCP](https://docs.gcp.databricks.com/getting-started/index.html).

* A workspace within the Databricks account for [AWS](https://docs.databricks.com/admin/workspace/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/admin/workspace/), or
  [GCP](https://docs.gcp.databricks.com/admin/workspace/index.html).

* The workspace's URL. Get the workspace URL for
  [AWS](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids),
  [Azure](https://learn.microsoft.com/azure/databricks/workspace/workspace-details#workspace-instance-names-urls-and-ids),
  or [GCP](https://docs.gcp.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids).

  Examples:

  * AWS: `https://<workspace-id>.cloud.databricks.com`
  * Azure: `https://adb-<workspace-id>.<random-number>.azuredatabricks.net`
  * GCP: `https://<workspace-id>.<random-number>.gcp.databricks.com`

  <Note>
    Do not add a trailing slash (`/`) to the workspace URL.
  </Note>

* The Databricks authentication details. For more information, see the documentation for
  [AWS](https://docs.databricks.com/dev-tools/auth/index.html),
  [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/),
  or [GCP](https://docs.gcp.databricks.com/dev-tools/auth/index.html).

  For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), the following Databricks authentication types are supported:

  * Databricks OAuth machine-to-machine (M2M) authentication for\
    [AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html).

    You will need the **Client ID** (or **UUID** or **Application** ID) and OAuth **Secret** (client secret) values for the corresponding service principal.
    Note that for Azure, only Databricks managed service principals are supported. Microsoft Entra ID managed service principals are not supported.

    The following video shows how to create a Databricks managed service principal:

    <iframe title="YouTube video player" />

  * Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html).

    You will need the personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  For [Unstructured Ingest](/open-source/ingestion/overview), the following Databricks authentication types are supported:

  * For Databricks personal access token authentication for
    [AWS](https://docs.databricks.com/dev-tools/auth/pat.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/pat), or
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/pat.html): The personal access token's value.

    The following video shows how to create a Databricks personal access token:

    <iframe title="YouTube video player" />

  * For username and password (basic) authentication ([AWS](https://docs.databricks.com/archive/dev-tools/basic.html) only): The user's name and password values.

  * For OAuth machine-to-machine (M2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-m2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-m2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-m2m.html)): The client ID and OAuth secret values for the corresponding service principal.

  * For OAuth user-to-machine (U2M) authentication ([AWS](https://docs.databricks.com/dev-tools/auth/oauth-u2m.html),
    [Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/oauth-u2m), and
    [GCP](https://docs.gcp.databricks.com/dev-tools/auth/oauth-u2m.html)): No additional values.

  * For Azure managed identities (formerly Managed Service Identities (MSI) authentication) ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-mi) only): The client ID value for the corresponding managed identity.

  * For Microsoft Entra ID service principal authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-sp) only): The tenant ID, client ID, and client secret values for the corresponding service principal.

  * For Azure CLI authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/auth/azure-cli) only): No additional values.

  * For Microsoft Entra ID user authentication ([Azure](https://learn.microsoft.com/azure/databricks/dev-tools/user-aad-token) only): The Entra ID token for the corresponding Entra ID user.

  * For Google Cloud Platform credentials authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-creds.html) only): The local path to the corresponding Google Cloud service account's credentials file.

  * For Google Cloud Platform ID authentication ([GCP](https://docs.gcp.databricks.com/dev-tools/auth/gcp-id.html) only): The Google Cloud service account's email address.

* The name of the parent catalog in Unity Catalog for
  [AWS](https://docs.databricks.com/catalogs/create-catalog.html),
  [Azure](https://learn.microsoft.com/azure/databricks/catalogs/create-catalog), or
  [GCP](https://docs.gcp.databricks.com/catalogs/create-catalog.html) for the volume.

* The name of the parent schema (formerly known as a database) in Unity Catalog for
  [AWS](https://docs.databricks.com/schemas/create-schema.html),
  [Azure](https://learn.microsoft.com/azure/databricks/schemas/create-schema), or
  [GCP](https://docs.gcp.databricks.com/schemas/create-schema.html) for the volume.

* The name of the volume in Unity Catalog for [AWS](https://docs.databricks.com/tables/managed.html),
  [Azure](https://learn.microsoft.com/azure/databricks/tables/managed), or
  [GCP](https://docs.gcp.databricks.com/tables/managed.html), and optionally any path in that volume that you want to access directly, beginning with the volume's root.

* The Databricks workspace user or service principal must have the following *minimum* set of privileges to read from or write to the
  existing volume in Unity Catalog:

  * `USE CATALOG` on the volume's parent catalog in Unity Catalog.
  * `USE SCHEMA` on the volume's parent schema (formerly known as a database) in Unity Catalog.
  * `READ VOLUME` and `WRITE VOLUME` on the volume.

  The following videos show how to create and set privileges for a catalog, schema (formerly known as a database), and volume in Unity Catalog.

  <iframe title="YouTube video player" />

  Learn more about how to check and set Unity Catalog privileges for
  [AWS](https://docs.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges),
  [Azure](https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/manage-privileges/#grant), or
  [GCP](https://docs.gcp.databricks.com/data-governance/unity-catalog/manage-privileges/index.html#show-grant-and-revoke-privileges).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Databricks Volumes**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Host** (*required*): The Databricks workspace host URL.

  <Note>
    Do not add a trailing slash (`/`) to the host URL.
  </Note>

* **Catalog** (*required*): The name of the catalog to use.

* **Schema** : The name of the associated schema. If not specified, **default** is used.

* **Volume** (*required*): The name of the associated volume.

* **Volume Path** : Any optional path to access within the volume.

* For **Authentication Method**, if you select **Service Principal**, you must also specify the following:

  * **OAuth Secret** (*required*): The associated OAuth **Secret** value for the Databricks managed service principal that has the appropriate privileges to the volume.
  * **UUID** (*required*): The **Client ID** (or **UUID** or **Application ID**) value for the Databricks managed service principal that has appropriate privileges to the volume.

* For **Authentication Method**, if you select **Token**, you must also specify the Databricks personal access token's value in the **Token** field.


# Dropbox
Source: https://docs.unstructured.io/pipelines/sources/dropbox



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Dropbox.

The requirements are as follows.

<iframe title="YouTube video player" />

1. A [Dropbox account](https://www.dropbox.com/try/teams).

2. A Dropbox app for your Dropbox account. To create a Dropbox app, do the following:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click **Create app**.<br />
   d) For **Choose an API**, select **Scoped access**.<br />
   e) For **Choose the type of access you need**, select **App folder**.<br />
   f) Enter a name for your app, and then click **Create app**.<br />
   g) On the app's **Permissions** tab, under **Files and folders**, check the boxes labelled **files.content.read** or **files.content.write** or both,
   depending on whether you want to read files, write files, or both. Then click **Submit**.<br />
   h) On the app's **Settings** tab, note the value of the **App folder name** field. This is the name of the app folder that Dropbox will create under the `Apps` top-level folder in your Dropbox
   account that the Dropbox app will use for access. If you change the value of **App folder name** field here, Dropbox will create an app folder with that name under the `Apps` top-level folder instead.<br />
   i) Under **OAuth 2**, next to **Generated access token**, click **Generate**. Copy the value of this access token. You should only click **Generate** after you have completed all of the preceding steps first.
   This is because the access token is scoped to the specific app folder and settings at the time the access token is generated. If you change the app folder name or any of the permissions later,
   you should regenerate the access token.<br />

   <Warning>
     Access tokens are valid for **only four hours** after they are created. After this four-hour period, you can no longer use the expired access token.
     Dropbox does not allow the creation of access tokens that are valid for more than four hours.

     To replace an expired access token, you must first generate a *refresh token* for the corresponding access token. To learn how to generate an access token and its corresponding refresh token,
     see [Replace an expired access token](#replace-an-expired-access-token), later in this article.

     If you do not already have the corresponding refresh token for an existing access token, or if you lose a refresh token after you generate it,
     you must generate a new access token and its corresponding refresh token.

     Instead of continualy replacing expired access tokens yourself, you can have Unstructured do it for you as needed; just supply Unstructured
     with the refresh token along with the Dropbox app's **App key** and **App secret** values.
     To learn how to supply these to Unstructured, look for mentions of "refresh token," "app key," and "app secret" in the connector settings later in this article.
   </Warning>

3. The app folder that your Dropbox app will use for access can be found in your Dropbox account under the `Apps` top-level folder. For example, if the value of the **App folder name**
   field above is `my-folder`, then the app folder that your Dropbox app will use for access can be found under `https://dropbox.com/home/Apps/my-folder`

   <img alt="The my-folder app folder under the Apps top-level folder" />

   <Warning>
     Your Dropbox app will *not* have access to upload or download files from the root of the app folder. Instead, you *must* create a subfolder inside of the app folder for your Dropbox
     app to upload or download files from. You will use the name of that subfolder when specifying your remote URL in the next step. For example, if your Dropbox app uses an app folder named `my-folder`
     for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then the subfolder that your Dropbox app will upload and download files from
     can be found under `https://dropbox.com/home/Apps/my-folder/data`

     <img alt="The data subfolder under the my-folder subfolder" />
   </Warning>

4. Note the remote URL to your subfolder inside of the app folder, which takes the format `dropbox://<subfolder-name>`. For example,
   if your Dropbox app uses an app folder named `my-folder` for access within the `Apps` top-level folder, and you create a subfolder named `data` within the `my-folder` app folder, then
   the remote URL is `dropbox://data`

   <img alt="The data subfolder under the my-folder subfolder" />

## Replace an expired access token

Dropbox app access tokens are valid for **only four hours**. After this time, you can no longer use the expired access token.

To have Unstructured automatically replace expired access tokens on your behalf, do the following:

<iframe title="YouTube video player" />

1. Get the app key and app secret values for your Dropbox app. To do this:

   a) Sign in to the [Dropbox Developers](https://www.dropbox.com/developers) portal with the same credentials as your Dropbox account.<br />
   b) Open your [App Console](https://www.dropbox.com/developers/apps).<br />
   c) Click your Dropbox app's icon.<br />
   d) On the **Settings** tab, next to **App key**, copy the value of the app key.<br />
   e) Next to **App secret**, click **Show**, and then copy the value of the app secret.

2. Use your web browser to browse to the following URL, replacing `<app-key>` with the app key for your Dropbox app:

   ```text theme={null}
   https://www.dropbox.com/oauth2/authorize?client_id=<app-key>&response_type=code&token_access_type=offline
   ```

3. Click **Continue**.

4. Click **Allow**.

5. In the **Access code generated** tile, copy the access code that is shown.

6. Use the [curl](https://curl.se/) utility in your Terminal or Command Prompt, or use a REST API client such as
   [Postman](https://www.postman.com/product/api-client/), to make the following REST API call, replacing the following placeholders:

   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.
   * Replace `<access-code>` with the access code that you just copied.

   ```text theme={null}
   curl --location --request POST 'https://api.dropbox.com/oauth2/token' \
   --user '<app-key>:<app-secret>' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data-urlencode 'code=<access-code>' \
   --data-urlencode 'grant_type=authorization_code'
   ```

7. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token.
   * The value of `refresh_token` is the refresh token that can be used to replace this access token much faster and easier next time.
     If you lose this refresh token, you must go back to Step 2.

   For the [Unstructured Pipelines](/pipelines/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the connector settings **Refresh token** field.
   * Add the `<app-key>` value to the connector settings **App key** field.
   * Add the `<app-secret>` value to the connector settings **App secret** field.

   For the [Unstructured API](/api-reference/overview) and [Unstructured Ingest](/open-source/ingestion/overview), if you want Unstructured to use this refresh token to automatically replace the expired access token instead of replacing it yourself, then
   add the following values to your connector settings, and then stop here:

   * Add the `refresh_token` value to the `refresh_token` parameter.
   * Add the `<app-key>` value to the `app_key` parameter.
   * Add the `<app-secret>` value to the connector settings `app_secret` parameter.

8. If for some reason you need to manually replace the expired access token yourself instead of having Unstructured do it for you, you can use the refresh token that you just copied to get a new access token:

   * Replace `<refresh-token>` with the refresh token.
   * Replace `<app-key>` with the app key for your Dropbox app.
   * Replace `<app-secret>` with the app secret for your Dropbox app.

   ```text theme={null}
   curl https://api.dropbox.com/oauth2/token \
   --data refresh_token=<refresh-token> \
   --data grant_type=refresh_token \
   --data client_id=<app-key> \
   --data client_secret=<app-secret>
   ```

9. In the response, copy the following two values:

   * The value of `access_token` (starting with the characters `sl`) is the new, valid access token. In the connector, replace the old,
     expired access token value with this new, valid access token value.
   * The value of `refresh_token` is the new, valid refresh token. To replace the expired access token yourself, go back to Step 8.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Dropbox**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Data URL** (*required*): The URL to the target subfolder inside of the app folder for the Dropbox app, starting with `dropbox://`.
* **Recursive** (source connector only): Check this box to also process all child folders and their files within the subfolder.
* **App key** (*required*) - The app key for your Dropbox app. This allows Unstructured to automatically replace expired access tokens.
* **Refresh token** (*required*) - The refresh token for the Dropbox app. This allows Unstructured to automatically replace expired access tokens.
* **App secret** (*required*) - The app secret for your Dropbox app. This allows Unstructured automatically to replace expired access tokens.


# Dropbox source connector quickstart
Source: https://docs.unstructured.io/pipelines/sources/dropbox-source-quickstart



Unstructured can connect to several types of [sources](/pipelines/connectors#sources). In this quickstart, you create a [Dropbox](https://www.dropbox.com/) source connector that you can add to your Unstructured [workflows](/pipelines/workflows).
This source connector enables your Unstructured workflows to process your files that you store in a Dropbox account.

In this quickstart, you will:

* Create a free Dropbox Basic account.
* Create a Dropbox app in your Dropbox account. This app will provide the connection between your Dropbox account and your Unstructured account.
* Get specific information about your Dropbox app that Unstructured needs to connect to the app.
* Create a Dropbox source connector in your Unstructured account.
* Add the Dropbox source connector to a workflow in your Unstructured account.

If you are not able to complete the following steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

## Step 1: Create your Dropbox account

In this step, you create a [free Dropbox Basic account](https://www.dropbox.com/basic). This account is limited to a single user and 2 GB of storage.

If you already have a Dropbox account and want to use it instead, then skip ahead to Step 2.

1. Go to the Dropbox account sign up page, at [https://www.dropbox.com/register](https://www.dropbox.com/register).
2. Enter your email address, and then click **Continue**.
3. Enter your name and then, for **Password**, enter some password for your new account. Be sure to save this password to some secure location,
   as you will need it to access your Dropbox account later.
4. Click **Agree and sign up**.
5. To create a Dropbox Basic account, click the **Personal** tile.
6. When you are prompted to install the Dropbox desktop app or mobile app, click **Skip step**. (You can always install these apps later.)
7. To create a Dropbox Basic account, click **Continue with 2 GB Dropbox Basic plan** at the bottom of the account type selectionpage.
8. You are automatically signed in to your new Dropbox Basic account, and the Dropbox user interface (UI) appears.

   <img alt="Dropbox user interface" />

## Step 2: Create a Dropbox app

In this step, you create a Dropbox app in your Dropbox account. Unstructured will use this app to access your Dropbox account.

1. From a new tab in your web browser, open the Dropbox Developers page, at [https://www.dropbox.com/developers](https://www.dropbox.com/developers).

2. Click **Create apps**.

   <img alt="Begin creating a Dropbox app" />

   <Tip>
     If **Create apps** is not visible then, on the top navigation bar, click **App Console**, and then click **Create app**.
   </Tip>

3. For **Choose an API**, select the **Scoped access** radio button.

4. For **Choose the type of access you need**, select the **App folder** radio button.

5. For **Name your app**, enter some name for your Dropbox app.

6. Check the box labelled **I agree to Dropbox API Terms and Conditions**, if it appears.

7. Click **Create app**.

   <img alt="Finish creating the Dropbox app" />

8. On the Dropbox app's **Permissions** tab, under **Files and folders**, check the box labelled **files.content.read**, and then click **Submit**.

   <img alt="Setting files.content.read for the Dropbox app" />

9. On the app's **Settings** tab, note the value of the **App folder name** field. This is the name of the subfolder that Dropbox will create under the `Apps` top-level folder in your Dropbox account. Your new Dropbox app will use this subfolder for access.

   <img alt="Noting the app folder's name" />

10. With the app's **Settings** tab still showing, scroll down to **App key**.

11. Next to **App secret**, click **Show**.

12. Note the values of **App key** and **App secret**, as you will need them later for Steps 3 and 5.

    <img alt="Noting the app's key and secret" />

## Step 3: Get a refresh token for your Dropbox app

In this step, you get a refresh token for your Dropbox app. Unstructured needs this refresh token, along with the
**App key** and **App secret** from the previous step, to be able to use your Dropbox app to connect to your Dropbox account.

1. In a new tab in your web browser, enter the following address. In this address,
   replace `<app-key>` with the **App key** you noted in Step 2:

   ```bash theme={null}
   https://www.dropbox.com/oauth2/authorize?client_id=<app-key>&response_type=code&token_access_type=offline
   ```

   For example, if your **App key** is `aaa1aaaaaa1aaaa`, then your address should look like this:

   ```bash theme={null}
   https://www.dropbox.com/oauth2/authorize?client_id=aaa1aaaaaa1aaaa&response_type=code&token_access_type=offline
   ```

2. Click **Continue**.

   <img alt="Continuing to get the access code" />

3. Click **Allow**.

   <img alt="Allowing the access code to be generated" />

4. Note the value in the **Access Code Generated** box.

   <img alt="Viewing the generated access code" />

   <Tip>
     This same access code value also appears in the **auth\_code** parameter from the address bar. For example, if the address bar shows this:

     ```bash theme={null}
     https://www.dropbox.com/oauth2/authorize_success?auth_code=ccc1ccc1ccc1-1cc-1cc&client_id=aaa1aaaaaa1aaaa&version=1
     ```

     Then the **auth\_code** value is `ccc1ccc1ccc1-1cc-1cc`.
   </Tip>

5. On your computer, open the Terminal if you're using macOS, a terminal window if you're using Linux, or the Command Prompt if you're using Windows.

6. Check to see if the `curl` utility is alredy installed on your computer by running the following command:

   ```bash theme={null}
   curl --version
   ```

7. If the `curl` utility is already installed, then you will see the `curl` version number and some other information about `curl`.

   If some kind of error message appears instead, then install `curl` on your computer by following [instructions for your operating system](https://ec.haxx.se/install/index.html).

8. From your Terminal, terminal window, or Command Prompt, run the following `curl` command. In this command,
   replace the following placeholders:

   * Replace `<app-key>` with the **App key** you noted in Step 2.
   * Replace `<app-secret>` with the **App secret** you noted in Step 2.
   * Replace `<access-code>` with the **Access Code Generated** (or `auth_code`, if you followed the Tip) that you noted earlier in this step.

   <Icon icon="apple" />  <Icon icon="linux" />  For macOS or Linux, run this `curl` command:

   ```bash theme={null}
   curl --location --request POST 'https://api.dropbox.com/oauth2/token' \
   --user '<app-key>:<app-secret>' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data-urlencode 'code=<access-code>' \
   --data-urlencode 'grant_type=authorization_code'
   ```

   <Icon icon="apple" />  <Icon icon="linux" />  For example, if your **App key** is `aaa1aaaaaa1aaaa`, your **App secret** is `bbb1bbb1bbb1bbb1`, and your **Access Code Generated** or `auth_code` value is `ccc1ccc1ccc1-1cc-1cc`, then your command should look like this:

   ```bash theme={null}
   curl --location --request POST 'https://api.dropbox.com/oauth2/token' \
   --user 'aaa1aaaaaa1aaaa:bbb1bbb1bbb1bbb1' \
   --header 'Content-Type: application/x-www-form-urlencoded' \
   --data-urlencode 'code=ccc1ccc1ccc1-1cc-1cc' \
   --data-urlencode 'grant_type=authorization_code'
   ```

   <Icon icon="windows" />  For Windows, run this `curl` command:

   ```bash theme={null}
   curl --location --request POST "https://api.dropbox.com/oauth2/token" ^
   --user "<app-key>:<app-secret>" ^
   --header "Content-Type: application/x-www-form-urlencoded" ^
   --data-urlencode "code=<access-code>" ^
   --data-urlencode "grant_type=authorization_code"
   ```

   <Icon icon="windows" />  For example, if your **App key** is `aaa1aaaaaa1aaaa`, your **App secret** is `bbb1bbb1bbb1bbb1`, and your **Access Code Generated** or `auth_code` value is `ccc1ccc1ccc1-1cc-1cc`, then your command should look like this:

   ```bash theme={null}
   curl --location --request POST "https://api.dropbox.com/oauth2/token" ^
   --user "aaa1aaaaaa1aaaa:bbb1bbb1bbb1bbb1" ^
   --header "Content-Type: application/x-www-form-urlencoded" ^
   --data-urlencode "code=ccc1ccc1ccc1-1cc-1cc" ^
   --data-urlencode "grant_type=authorization_code"
   ```

9. In the response, note the vaue of the `refresh_token` field, which you will need later for Step 5. For example, if the response looks like this
   (line breaks are added here for readability—the actual response will not have these line breaks):

   ```bash theme={null}
   {
       "access_token": "sl.u.AF-aaa1aaa1", 
       "token_type": "bearer", 
       "expires_in": 14400, 
       "refresh_token": "ddd1ddd1ddd1ddd1", <-- Note this value.
       "scope": "account_info.read files.content.read files.metadata.read", 
       "uid": "1111111111", 
       "account_id": "dbid:aaa1aaa1aaa1"
   }
   ```

   Then the value of the `refresh_token` field is `ddd1ddd1ddd1ddd1`.

## Step 4: Upload documents to your Dropbox app folder

In this step, you add your documents to your Dropbox app folder in your Dropbox account.

1. In the Dropbox UI from Step 1, expand `Apps`, and then click the folder that matches the **App folder name** field you noted in Step 2.

   <img alt="Selecting the app folder" />

   <Tip>
     If the `Apps` folder is not clickable, or if Dropbox reports that the folder is not found,
     then try refreshing the page and clicking the `Apps` folder again.
   </Tip>

2. Click **Create folder**.

   <img alt="Begin creating the subfolder" />

3. Give the subfolder a name, and then click **Create**.

   <img alt="Finish creating the subfolder" />

4. Click **Upload or drop** (or **Upload > Files** or **Upload > Folder**), and then follow the on-screen instructions to upload some documents to this subfolder in your Dropbox app folder.
   For a Dropbox Basic account, the total size of all of the files you upload and store in your Dropbox account (not just this subfolder) cannot exceed 2 GB.

   <img alt="Upload files to subfolder" />

## Step 5: Create the Dropbox source connector

In this step, you create a Dropbox source connector in your Unstructured account. This source connector
is used by Unstructured to connect to your Dropbox account and then process the documents in the specified folder.

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/pipelines/overview#how-am-i-billed%3F).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. On the sidebar, click **Connectors**.

   <img alt="Connectors sidebar icon" />

4. Click **+ New**.

5. Enter some unique name for this connector, for example `dropbox-source-connector`.

6. For **Type**, click **Source**.

7. For **Provider**, click **Dropbox**.

8. Click **Continue**.

   <img alt="Selecting the connector type" />

9. For **Data URL**, enter `dropbox://`, followed by the name of the subfolder you created in the previous step. For example,
   if the name of the subfolder is `my-folder`, then the data URL would be `dropbox://my-folder`.

   <img alt="Forming the data URL" />

10. For **App key**, enter the **App key** you noted in Step 2.

11. For **App secret**, enter the **App secret** you noted in Step 2.

12. For **Refresh token**, enter the **Refresh token** you noted in Step 3.

13. Click **Save and Test**, and wait while Unstructured tests the connector.

<img alt="Entering connector settings" />

14. If a green **Successful** message appears, then you have successfully created the connector.

    <img alt="Successful connector test" />

    If, however, a red error message appears, fix the issue, and try this step again.

    If you cannot fix the issue, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

Congratulations! You have successfully created a Dropbox source connector in your Unstructured account.

If you are not able to complete these steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

## Next steps

* If you do not have a destination connector in your Unstructured account, then complete the [Pinecone destination connector quickstart](/pipelines/destinations/pinecone-destination-quickstart).

  If you're not sure if you have a destination connector, click **Connectors** in your Unstructured account's sidebar, and then click **Destinations** to see if there are any listed.

  <img alt="Connectors sidebar icon" />

* If you already have a destination connector, then you can add this Dropbox source connector as well as your destination connector to a workflow in your Unstructured account. To do this:

  1. Click **Workflows** in your Unstructured account's sidebar.

     <img alt="Workflows sidebar icon" />

  2. Click **New Workflow +**.

     <img alt="Begin creating a workflow" />

  3. With **Build it Myself** already selected, click **Continue**.

     <img alt="Build it myself workflow type" />

  4. Click the **Source** node. (Do not click **Drop file to test**.)

     <img alt="Workflow source node" />

  5. On the **Details** tab, click **Connectors**, and then click the name of your Dropbox source connector.

     <img alt="Selecting the source connector" />

  6. Click the **Destination** node.

     <img alt="Workflow destination node" />

  7. On the **Details** tab, click the name of your destination connector.

     <img alt="Selecting the destination connector" />

  8. Switch **Active** to on, and then click **Save**.

     <img alt="Making the workflow active" />

  9. Next to your workflow's name, click **Run**.

     <img alt="Running the workflow" />

  10. Click **Jobs** in your Unstructured account's sidebar.

      <img alt="Jobs sidebar icon" />

  11. After the job shows **Finished** with a green checkmark, go to your destination's location to see Unstructured's processed data output.

  <img alt="Finished job status" />

If you are not able to complete these steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# Elasticsearch
Source: https://docs.unstructured.io/pipelines/sources/elasticsearch



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Elasticsearch.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), only Elastic Cloud instances are supported.

* For [Unstructured Ingest](/open-source/ingestion/overview), Elastic Cloud instances and self-manged Elasticsearch instances are supported.

* For Elastic Cloud, you will need an [Elastic Cloud service instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#hosted-elasticsearch-service).

  <iframe title="YouTube video player" />

* For self-managed Elasticsearch, you will need a [self-managed Elasticsearch instance](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html#elasticsearch-deployment-options).

  <iframe title="YouTube video player" />

* You will need the name of the index on the instance.

  For the destination connector, if you need to create an index, you can use for example the following `curl` command. Replace the following placeholders:

  * Replace `<host>:<port>` with the instance's host identifier and port number.

  * Replace one of the following, and remove the other:

    * For an instance's user and password combination, replace `<username>` with your Elasticsearch or Elastic Cloud instance's user name, replace `<password>` with your password, and then remove `--user "Authorization: ApiKey <your-api-key>"`.
    * For an Elastic Cloud API key, replace `<your-api-key>` with your Elastic Cloud API key, and then remove `--user "<username>:<password>"`.

  * Replace `<index-name>` with the name of the new index on the instance.

  * Replace `<index-schema>` with the schema for the index. A schema is optional; see the explanation
    following this `curl` command for more information.

  ```bash theme={null}
  curl --request PUT "<host>:<port>/<index-name>" \
  --user "<username>:<password>" \
  --user "Authorization: ApiKey <your-api-key>" \
  [--header "Content-Type: application/json" \
  --data '<index-schema>']
  ```

  To learn more, see [Create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) and [Get index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-index.html).

  For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
  Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
  to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  For objects in the `metadata` field that Unstructured produces and that you want to store in an Elasticsearch or Elastic Cloud index, you must create fields in your index's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    },
    "entities": {
      "items": [
        {
          "entity": "HOME FOR FUTURE",
          "type": "ORGANIZATION"
        },
        {
          "entity": "221 Queen Street, Melbourne VIC 3000",
          "type": "LOCATION"
        }
      ],
      "relationships": [
        {
          "from": "HOME FOR FUTURE",
          "relationship": "based_in",
          "to": "221 Queen Street, Melbourne VIC 3000"
        }
      ]
    }
  }
  ```

  You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
  required by Unstructured whenever you create your own index schema:

  * `element_id`
  * `record_id`, which is required by Unstructured for intelligent record updates.
  * `type`, which is not required, but highly recommended.
  * `text`
  * `embeddings` if embeddings are generated; make sure to set `dims` to the same number of dimensions as the embedding model generates.

  ```json theme={null}
  {
    "mappings": {
      "properties": {
        "element_id": {
          "type": "keyword"
        },
        "record_id": {
          "type": "text"
        },
        "text": {
          "type": "text"
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "embeddings": {
          "type": "dense_vector",
          "dims": 1536,
          "index": true,
          "similarity": "cosine"
        },
        "metadata": {
          "properties": {
            "is_extracted": {
              "type": "boolean"
            },
            "coordinates-points": {
              "type": "float"
            },
            "coordinates-system": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "coordinates-layout_width": {
              "type": "long"
            },
            "coordinates-layout_height": {
              "type": "long"
            },
            "filetype": {
              "type": "keyword"
            },
            "languages": {
              "type": "keyword"
            },
            "page_number": {
              "type": "integer"
            },
            "image_mime_type": {
              "type": "keyword"
            },
            "filename": {
              "type": "keyword"
            },
            "data_source-url": {
              "type": "keyword"
            },
            "data_source-record_locator-protocol": {
              "type": "keyword"
            },
            "data_source-record_locator-remote_file_path": {
              "type": "keyword"
            },
            "entities-items": {
              "properties": {
                "entity": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "entities-relationships": {
              "properties": {
                "from": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "relationship": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "to": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
  ```

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  See also:

  * [An Introduction to Elasticsearch Mapping](https://www.elastic.co/blog/found-elasticsearch-mapping-introduction)
  * [Explicit mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html)
  * [Dynamic field mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html)
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  <iframe title="YouTube video player" />

* For Elastic Cloud, you will need the Elastic Cloud service instance's API key. If you are using Unstructured Ingest, you will also need the instance's Cloud ID. To get these, see your Elasticsearch Service web console or [Elastic Cloud API keys](https://www.elastic.co/docs/deploy-manage/api-keys/elastic-cloud-api-keys).

  <Note>
    When generating an Elastic Cloud service instance's API key through the Elasticsearch Service web console, be sure after you click **Create API key** to select **Encoded** to get the API key's value.
    (The **Encoded** API key value is the Base64 encoding of the UTF-8 representation of the API key's ID and key value joined by a colon \[:].)
  </Note>

* For self-managed Elasticsearch, you will need:

  * The self-managed instance's hostname and port number. See [Networking](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html).

  * If you're using basic authentication to the self-managed instance, the user's name and password.

  * If you're using token-based authentication to the self-managed instance, the bearer token or API key for the instance. See [Token-based authentication services](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-authentication-services.html) and [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

    <Note>
      When generating an API key for a self-managed Elasticsearch instance by calling [POST /\_security/api\_key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key), be sure to use
      the `encoded` value in the response payload. (This `encoded` value is the Base64 encoding of the UTF-8 representation of the payload's `id` and `api_key` values joined by a colon \[:]). Do not use the `api_key` value by itself.
    </Note>

  * If you're using certificate authentication to the self-managed instance, the path to the Certificate Authority (CA) file on the instance, and the certificate fingerprint. See [SSL certificate API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-ssl.html) and [Where can I see my Certificate Fingerprint?](https://discuss.elastic.co/t/where-can-i-see-my-certificate-fingerprint/319335/3).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Elasticsearch**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host** (*required*): The endpoint URL for the target cluster.
* **Index Name** (*required*): The name of the target index in the cluster.
* **API Key** (*required*): The Elastic Cloud API key for the target cluster.


# IBM FileNet
Source: https://docs.unstructured.io/pipelines/sources/filenet

Connect Unstructured to IBM FileNet as a source to ingest documents and content from your FileNet content repository into Unstructured.

IBM FileNet is an enterprise content management system that enables organizations to store, manage, secure, and automate documents and business content at scale.

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* An [IBM Cloud Pak for Business Automation as a Service](https://www.ibm.com/products/cloud-pak-for-business-automation) account with access to the IBM FileNet [Content Platform Engine server](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=architecture-content-platform-engine) object store to which you want to connect. IBM FileNet is a component of IBM Cloud Pak for Business Automation as a Service.

To access the information you'll need to configure the connector:

1. Log into your [account](https://www.automationcloud.ibm.com/auth/index.jsp).

2. Choose the **Navigator** tile.

   The [IBM Navigator](https://www.ibm.com/docs/en/content-navigator/3.2.0) launches in a separate browser window, and provides a view of your object stores and content. You can use the IBM Navigator views to find the information necessary to create a connection to Unstructured.

   For the URL of your IBM FileNet server:

   * The server URL displays in the browser address bar. You only need the base URL that specifies the company and domain. For example, `https://<company-name>.automationcloud.ibm.com`.

   For [object store](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=infrastructure-defining-object-stores) names and folder paths:

   * Select the folder in the left pane. The full folder path is displayed at the top of the main detail pane, in the following format: `<object-store>/<folder>/etc`.

   For the document class:

   * Right-click the document and select **Properties**.

   For the account username:

   * Right-click the profile icon on the upper right in the top menu.

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document. It also lists any users or groups that are explicitly *denied* those permissions.

For more information on how IBM FileNet uses Access Control Lists (ACLs) and Access Control Entries (ACEs) to manage permissions, see [About access rights](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=authorization-about-access-rights) in the IBM FileNet Platform documentation.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

<Warning>
  Always take into account the `deny_users` and `deny_groups` fields when determining the effective permissions a user or group has. Not doing so may result in over-granting permissions.

  Unstructured does not resolve groups into users when outputting permissions metadata.
</Warning>

Unstructured derives permissions metadata from the document's ACL, returned inline with the document's metadata via the IBM FileNet GraphQL API.

The document ACL returned by IBM FileNet includes the full effective ACL, including inherited permissions. Because of this, Unstructured does not further query for inherited permissions. For more information, see [ACE source: Default, Direct, Inherited, Template](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=rights-ace-source-default-direct-inherited-template) in the IBM FileNet Platform documentation.

Unstructured does not include the following permission values in the `permissions_data` field:

* MARKING

  Marking is a security classification system layered on top of standard ACL permissions. For more information, see [Markings overview](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=markings-overview) in the IBM FileNet Platform documentation.
* PROXY

  Permissions granted to a principal to act on behalf of another principal.

Unstructured writes a maximum of 1000 permission entries (ALLOW and DENY) per file.

### Permissions evaluation

IBM FileNet supports explicit DENY ACEs and evaluates them with DENY-wins semantics. This means that if a user has both an ALLOW and a DENY ACE for the same action, whether directly or through group membership, the DENY takes
precedence. For more information, see [Allow or Deny and order of evaluation](https://www.ibm.com/docs/en/filenet-p8-platform/5.7.0?topic=permissions-allow-or-deny-and-order-of-evaluation) in the IBM FileNet Platform documentation.

### Identifier formats

The connector takes the `granteeName` value that IBM FileNet returns and writes it directly into `users`, `groups`, `deny_users`, or `deny_groups` without any modification. The format of that identifier depends entirely on how your specific IBM FileNet instance is connected to its directory service.

* IBM FileNet SaaS (IBM Cloud Identity)

  LDAP distinguished names are used. For example:

  Users:`uid=alice.smith,cn=users,O=IBM,C=US`

  Groups: `cn=Finance,cn=groups,O=IBM,C=US`

* On-premises with Active Directory

  Several possible formats are possible. For example:

  * A CN=.../DC=... distinguished name:

    `CN=Alice Smith,OU=Staff,DC=contoso,DC=com`

  * A DOMAIN\user short name:

    `CONTOSO\alice.smith`

  * A Windows SID:

    `S-1-5-21-3623811015-3361044348-30300820-1013`

  * A userPrincipalName:

    `alice.smith@contoso.com`

If you're building a
downstream system that filters on these values, you will need to verify which format your instance
uses before writing that logic. We recommend running a sample query against your instance to check.

### Metadata output example

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity. This example uses IBM FileNet SaaS (IBM Cloud Identity) with LDAP distinguished names.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": ["uid=alice,cn=users,O=IBM,C=US"],
                            "groups": ["cn=Finance,cn=groups,O=IBM,C=US"],
                            "deny_users": ["uid=contractor,cn=users,O=IBM,C=US"],
                            "deny_groups": []
                        }
                    },
                    {
                        "update": {
                            "users": ["uid=alice,cn=users,O=IBM,C=US"],
                            "groups": [],
                            "deny_users": [],
                            "deny_groups": []
                        }
                    },
                    {
                        "delete": {
                            "users": [],
                            "groups": [],
                            "deny_users": [],
                            "deny_groups": []
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

## Creating the connector

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Click **New** or **Create Connector**.
4. For **Name**, enter a unique name for this connector.
5. In the **Provider** area, click **IBM FileNet**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Server URL** (*required*): The base URL of your Content Platform Engine, containing both the IBM domain and your company's subdomain. For example,  `https://<company-name>.automationcloud.ibm.com`.
* **Object Store** (*required*): The name of the object store to connect within the server.
* **Folder Path** (*required*, source connector only): The path to the folder within the object store to use as the source.
* **Target Folder** (*required*, destination connector only): The path to the folder within the object store to use as the upload destination.
* **Document Class**: The class of documents to include.
* **Recursive**: (Source connector only) Select to include documents contained in any subfolders.
* **Username** (*required*): The username of the IBM Cloud Pak for Business Automation as a Service account to use.
* **Password** (*required*): The password for the corresponding username.


# Google Cloud Storage
Source: https://docs.unstructured.io/pipelines/sources/google-cloud



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Google Cloud Storage.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Google Cloud service account. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#console).

* A service account key for the service account. See [Create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating) in
  [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the service account key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:
    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```
  * For Windows:
    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* The URI for a Google Cloud Storage bucket. This URI consists of the target bucket name, plus any target folder within the bucket, expressed as `gs://<bucket-name>[/folder-name]`. [Create a bucket](https://cloud.google.com/storage/docs/creating-buckets#console).

  This bucket must have, at minimum, one of the following roles applied to the target Google Cloud service account:

  * `Storage Object Viewer` for bucket read access.
  * `Storage Object User` for bucket write access.
  * The `Storage Object Admin` role provides read and write access, plus access to additional bucket operations.

  To apply one of these roles to a service account for a bucket, see [Add a principal to a bucket-level policy](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#bucket-add)
  in [Set and manage IAM policies on buckets](https://cloud.google.com/storage/docs/access-control/using-iam-permissions).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Google GCS**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bucket URI** (*required*): The URI for the Google Cloud Storage bucket and any target folder path within the bucket. This URI takes the format `gs://<bucket-name>[/folder-name]`.
* **Recursive** (source connector only): Check this box to ingest data recursively from any subfolders, starting from the path specified by **Bucket URI**.
* **Service Account Key** (*required*): The contents of a service account key file, expressed as a single string without line breaks, for a Google Cloud service account that has the required access permissions to the bucket.


# Google Drive
Source: https://docs.unstructured.io/pipelines/sources/google-drive



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Google Drive.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Google Cloud account](https://console.cloud.google.com).

* The **Google Drive API** enabled in the account.
  [Learn how](https://cloud.google.com/apis/docs/getting-started#enabling_apis).

* Within the account, a Google Cloud service account and its related `credentials.json` key file or its contents in JSON format.
  [Create a service account](https://developers.google.com/workspace/guides/create-credentials#create_a_service_account).
  [Create credentials for a service account](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account).

  To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as
  a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself).
  To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-downloaded-key-file>` with the path to the `credentials.json` key file that you downloaded by following the preceding instructions.

  * For macOS or Linux:

    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```

  * For Windows:

    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* A Google Drive [shared folder](https://support.google.com/drive/answer/2375091) or [shared drive](https://support.google.com/a/users/answer/7212025).

* Give the service account access to the shared folder or shared drive. To do this, share the folder or drive with the service account's email address.
  [Learn how](https://support.google.com/drive/answer/7166529).
  [Learn more](https://www.googlecloudcommunity.com/gc/Workspace-Q-A/Can-i-give-access-to-document-of-google-drive-to-service-account/m-p/530106).

* Get the shared folder's ID or shared drive's ID. This is a part of the URL for your Google Drive shared folder or shared drive, represented in the following URL as `{folder_id}`: `https://drive.google.com/drive/folders/{folder-id}`.

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "11111111111111111111"
                            ],
                            "groups": [
                                "22222222222222222222",
                                "33333333333333333333"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "44444444444444444444",
                                "55555555555555555555"
                            ],
                            "groups": [
                                "66666666666666666666",
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "77777777777777777777"
                            ],
                            "groups": [
                                "88888888888888888888"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Google Cloud user, use the user's ID along with the
[Admin SDK API](https://developers.google.com/workspace/admin/directory/reference/rest/v1/users/get) or the
[People API](https://developers.google.com/people/api/rest/v1/people/get) for Google Cloud.

To look up information about a particular Google Cloud group, use the group's ID along with the
[Admin SDK API](https://developers.google.com/workspace/admin/directory/reference/rest/v1/groups/get) or the
[Cloud Identity API](https://cloud.google.com/identity/docs/reference/rest/v1/groups/get) for Google Cloud.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Google Drive**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Drive ID** (*required*): The target folder's or drive's ID.

* **Extensions**: A comma-separated list of any file extensions to be included in the ingestion process (such as `jpg,pdf`), if filtering is needed.
  The default is to include all files, if not otherwise specified.

  <Note>
    Do not include the leading dot in the file extensions. For example, use `jpg` or `pdf` instead of `.jpg` or `.pdf`.
  </Note>

* **Recursive**: Check this box to also access files from all subfolders within the folder or drive.

* **Account Key** (*required*): The contents of the `credentials.json` key file for the target service account. These contents must be expressed as a single-line string without line breaks.


# Jira
Source: https://docs.unstructured.io/pipelines/sources/jira



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Jira.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Jira Cloud account](https://www.atlassian.com/try/cloud/signup?bundle=jira-software\&edition=free) or
  [Jira Data Center installation](https://confluence.atlassian.com/adminjiraserver/installing-jira-data-center-938846870.html).
* The site URL for your [Jira Data Center installation](https://confluence.atlassian.com/jirakb/find-your-site-url-to-set-up-the-jira-data-center-and-server-mobile-app-954244798.html) or Jira Cloud account.
  For Jira Cloud, open Jira in your web browser and copy the address from the browser's address bar.
  If you're unsure, check the dashboard URL, or if viewing an issue, project or board, the site URL is typically everything that comes before **but not including** `/jira`, such as
  `https://<organization>.atlassian.net`.
* To process Jira projects, provide the IDs for the target projects. To get a project's ID, sign in to your Jira Cloud account or Jira Data Center installation, and then go to the following URL: `https://<organization>.atlassian.net/rest/api/latest/project/<project-key>`,
  replacing `<organization>` with yours, and replacing `<project-key>` with the target project's key. In the
  response, look for the URL `https://<organization>.atlassian.net/rest/api/3/project/<project-id>`, where `<project-id>` is the target project's ID.
* To process Jira boards, the IDs for the target boards. To get a board's ID, sign in to your Jira Cloud account or Jira Data Center installation, and then go to the following URL: `https://<organization>.atlassian.net/rest/agile/1.0/board?projectKeyOrId=<project-key-or-id>`,
  replacing `<organization>` with yours, and `<project-key-or-id>` with the associated project's key or ID. In the
  response, look for the URL `https://<organization>.atlassian.net/rest/agile/1.0/board/<board-id>`, where `<board-id>` is the board's ID.
* To process Jira issues, the IDs for the target issues. To get an issue's ID, sign in to your Jia Cloud account or Jira Data Center installation, open the issue, and then look at the URL in your browser's address bar. The issue ID is the string of characters after the final slash in the URL.
* A user in your [Jira Cloud account](https://support.atlassian.com/jira-cloud-administration/docs/manage-users-groups-permissions-and-roles-in-jira-cloud/) or
  [Jira Data Center installation](https://confluence.atlassian.com/adminjiraserver/create-edit-or-remove-a-user-938847025.html).
* The user must have the correct permissions in your
  [Jira Cloud account](https://support.atlassian.com/jira-cloud-administration/docs/manage-users-groups-permissions-and-roles-in-jira-cloud/) or
  [Jira Data Center installation](https://confluence.atlassian.com/jirakb/permissions-made-simple-for-jira-server-717062767.html) to
  access the target projects, boards, and issues.
* One of the following:

  * For Jira Cloud or Jira Data Center, the target user's name or email address, and password.
    [Change a Jira Cloud user's password](https://support.atlassian.com/user-management/docs/change-password-for-portal-only-customers/).
    [Change a Jira Data Center user's password](https://confluence.atlassian.com/adminjiraserver/create-edit-or-remove-a-user-938847025.html).
  * For Jira Cloud only, the target user's name or email address, and API token.
    [Create an API token](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/).
  * For Jira Data Center only, the target user's personal access token (PAT).
    [Create a PAT](https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Jira**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **URL** (*required*): The URL of the Jira instance.
* **Username** (*required* for password or API token authentication): The username of the Jira user.
* **Password** (*required* for password or API token authentication): For password authentication, the password of the Jira user.
  For API token authentication, the API token of the Jira user.
* **Personal Access Token** (*required* for personal access token authentication): The personal access token of the Jira user.
* **Cloud**: Check this box if you are using Jira Cloud. The default is unchecked to use Jira Data Center.
* **Projects**: A comma-separated list of IDs of the target projects in Jira to access.
* **Boards**: A comma-separated list of IDs of the target boards in Jira to access.
* **Issues**: A comma-separated list of IDs of the target issues in Jira to access.
* **Status Filter**: A comma-separated list of statuses to filter Jira issues by.
* **Download Attachments**: If checked, download attachments from Jira issues. By default, attachments are not downloaded.

## Learn more

* <Icon icon="blog" />  [Jira Integration in Unstructured: Build RAG Systems with Project Management Data](https://unstructured.io/blog/jira-integration-in-unstructured-build-rag-systems-with-project-management-data)


# Kafka
Source: https://docs.unstructured.io/pipelines/sources/kafka



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Kafka.

The requirements are as follows.

* A Kafka cluster in [Confluent Cloud](https://www.confluent.io/confluent-cloud).
  ([Create a cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#create-ak-clusters).)

  The following video shows how to set up a Kafka cluster in Confluent Cloud:

  <iframe title="YouTube video player" />

* The [hostname and port number](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#view-a-ak-cluster) of the bootstrap Kafka cluster to connect to..

* The name of the topic to read messages from or write messages to on the cluster.
  [Create a topic](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).
  [Access available topics](https://docs.confluent.io/cloud/current/client-apps/topics/index.html#create-topics).

* For authentication, an [API key and secret](https://docs.confluent.io/cloud/current/security/authenticate/workload-identities/service-accounts/api-keys/manage-api-keys.html#add-an-api-key).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Kafka**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bootstrap Server** (*required*): The hostname of the bootstrap Kafka cluster to connect to.
* **Port**: The port number of the cluster.
* **Group ID**: The ID of the consumer group, if any, that is associated with the target Kafka cluster.
  (A consumer group is a way to allow a pool of consumers to divide the consumption of data
  over topics and partitions.) The default is `default_group_id` if not otherwise specified.
* **Topic** (*required*): The unique name of the topic to read messages from and write messages to on the cluster.
* **Number of messages to consume**: The maximum number of messages to get from the topic. The default is `100` if not otherwise specified.
* **Batch Size**: The maximum number of messages to send in a single batch. The default is `100` if not otherwise specified.
* **API Key** (*required*): The Kafka API key value.
* **Secret** (*required*): The secret value for the Kafka API key.

## Learn more

* <Icon icon="blog" />  [Unstructured Platform Now Integrates with Apache Kafka in Confluent Cloud](https://unstructured.io/blog/unstructured-platform-now-integrates-with-apache-kafka-in-confluent-cloud)


# MongoDB
Source: https://docs.unstructured.io/pipelines/sources/mongodb



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from MongoDB.

The requirements are as follows.

<iframe title="YouTube video player" />

The MongoDB requirements for a MongoDB Atlas deployment include:

<Warning>
  For MongoDB Atlas, SCRAM-SHA-1 is not supported for authentication. This means that cluster types that only
  include SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, and **Serverless**, are **not** supported.
  Unstructured only supports SCRAM-SHA-256 for MongoDB Atlas, which is cryptographically stronger than SCRAM-SHA-1.

  If you try to test or use a connector that refers to a cluster type that only includes SCRAM-SHA-1, the
  operation will fail, and you will get an error message similar to the following:
  `[digital envelope routines] unsupported`.
</Warning>

* A MongoDB Atlas account. [Create an account](https://www.mongodb.com/cloud/atlas/register).

* A MongoDB Atlas cluster. [Create a cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). Be sure to **not**
  select a cluster type that only includes SCRAM-SHA-1, such as **Free**, **M0**, **Flex**, or **Serverless**.

* The cluster must be reachable from your application environment, for example by adding IP addresses to your IP access list. [Learn more](https://www.mongodb.com/docs/atlas/setup-cluster-security/#network-and-firewall-requirements).

* The cluster must be configured to allow IP address. [Learn how](https://www.mongodb.com/docs/atlas/security/ip-access-list/#add-ip-access-list-entries).

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

* The cluster must have at least one database. [Create a database](https://www.mongodb.com/docs/compass/current/databases/#create-a-database).

* The database must have at least one user, and that user must have sufficient access to the database. [Create a database user](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#add-database-users). [Give the user database access](https://www.mongodb.com/docs/manual/core/authorization/).

* The database must have at least one collection. [Create a collection](https://www.mongodb.com/docs/compass/current/collections/#create-a-collection).

  <Note>
    For the destination connector, Unstructured recommends that all documents in the target collection have a field
    named `record_id` with a `String` data type.
    Unstructured can use this field to do intelligent document overwrites. Without this field, duplicate documents
    might be written to the collection or, in some cases, the operation could fail altogether.
  </Note>

* The connection string for the cluster. For MongoDB Atlas, this connection string must include the protocol, username, password, host, and cluster name. For example:

  ```text theme={null}
  mongodb+srv://<db_user>:<db_password>@<host>/?retryWrites=true&w=majority&appName=<cluster>
  ```

  To get the connection string in MongoDB Atlas, do the following:

  1. Log in to your MongoDB Atlas console.
  2. In the sidebar, under **Databases**, click **Clusters**.
  3. Click on the cluster you want to connect to.
  4. Click **Connect**.
  5. Click **Drivers**.
  6. Under **Add your connection string into your application code**, copy the connection string.
     You can then close the **Connect** dialog in MongoDB Atlas.

     Before you use this connection string, be sure to fill in any placeholders in the string, such as your MongoDB Atlas database user's password value.

  [Learn more](https://www.mongodb.com/resources/products/fundamentals/mongodb-connection-string).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **MongoDB**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Database** (*required*): The name of the database on the instance.
* **Collection** (*required*): The name of the collection within the database.
* **Connection String** (*required*): The MongoDB instance connection string.


# OneDrive
Source: https://docs.unstructured.io/pipelines/sources/onedrive



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from OneDrive.

The requirements are as follows.

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  (Even if you only plan to use OneDrive, you still need a plan that includes SharePoint Online, because OneDrive is built on SharePoint technology.)
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* The OneDrive and SharePoint Online plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The path to the target OneDrive folder, starting from the OneDrive account's root folder, for example `my-folder/my-subfolder`.

  The following video shows how to get a path:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials, and a username and password. Both authentication types require a Microsoft Entra ID app registration.

  <Note>
    The OneDrive connector does not support any other authentication methods, such as multifactor (MFA) or passwordless authentication.
  </Note>

  You will need to provide the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration that has access to the target OneDrive account, and
  the app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Files.ReadWrite.All` (if both reading and writing are needed)
  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
  * `Directory.Read.All`

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* Both authentication types also require the User Principal Name (UPN) for the OneDrive account. This is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

* For username and password authentication, you will also need the password for the target UPN.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **OneDrive**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Client ID** (*required*): The application (client) ID of the Microsoft Entra ID app registration that has access to the OneDrive account.
* **Tenant ID** (*required*): The directory (tenant) ID of the Entra ID app registration.
* **Authority URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
* **Principal Name** (*required*): The User Principal Name (UPN) for the OneDrive user account in Entra ID. This is typically the user's email address.
* **Client Credential** (*required*): The client secret for the Entra ID app registration.
* **Password** (*required* for username and password authentication): The password for the target UPN.
* **Path** (source connector only): The path to the target folder in the OneDrive account, starting with the account's root folder, for example `my-folder/my-subfolder`.
* **Recursive** (source connector only): Check this box to recursively access files from subfolders within the specified OneDrive path.
* **Remote URL** (destination connector only): `onedrive://`, followed by the path to the target folder in the OneDrive account, starting with the account's root folder, for example `onedrive://my-folder/my-subfolder`.


# OpenSearch
Source: https://docs.unstructured.io/pipelines/sources/opensearch



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from OpenSearch.

## Set up OpenSearch

Supported OpenSearch installations vary by product:

* **[Unstructured Pipelines](/pipelines/overview) and [Unstructured API](/api-reference/overview):** Non-local OpenSearch only. Local OpenSearch instances are not supported.
* **[Unstructured Ingest](/open-source/ingestion/overview):** Both local and non-local OpenSearch instances are supported.

### Set up an AWS OpenSearch Service domain

To set up an [AWS OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) domain, complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.

4. Click **Create domain**.

5. In the **Name** tile, for **Domain name**, enter some unique domain name for your new OpenSearch domain.

6. In the **Domain creation method** tile, select **Easy create**. This option provides faster setup using default configurations and enables fine-grained access control (FGAC) by default. (With **Standard create**, you must enable FGAC manually.) [Standard create method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html).

7. In the **Engine options** tile, for **Version**, AWS recommends that you select the latest version.

8. In the **Network** tile, for **Network**, select a network access method.
   For faster setup, this example uses the **Public access** method.
   [VPC access method documentation](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html#prerequisites-vpc-endpoints).

9. For **IP address type**, select **Dual-stack mode**.

10. In the **Fine-grained access control** (FGAC) tile, do one of the following:

    * If you want to use an existing AWS IAM user in the AWS account as the domain's master user, then for **Master user**, select **Set IAM ARN as master user**. Then enter the IAM ARN for the master user in the **IAM ARN** box.
    * If you want to create a master user and password as the domain's master user instead, then for **Master user**, select **Create master user**. Then specify some username and password for this
      new master user by filling in the **Master username**, **Master password**, and **Confirm master password** fields. Make
      sure to save the master user's password in a secure location.

11. Click **Create**.

12. After the domain is created, you must allow Unstructured to access the domain, as follows:

    a. If the new domain's settings page is not already showing, open it as follows:
    in your Amazon OpenSearch Service console, on the sidebar, expand **Managed clusters**, and then click **Domains**. Then,
    in the list of available domains, click the name of the newly created domain.<br />
    b. On the **Security configuration** tab, click **Edit**.<br />
    c. In the **Access policy** tile, for **Domain access policy**, select **Only use fine-grained access control**.<br />
    d. Click **Clear policy**. This removes any existing resource-based access policy from the domain. With no domain access policy, access control relies entirely on fine-grained access control (FGAC).<br />
    e. Click **Save changes**.

### Set up an Amazon OpenSearch Serverless collection

To set up an [Amazon OpenSearch Serverless collection](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-create-console.html), complete steps similar to the following:

1. Sign in to your AWS account, and then open your AWS Management Console.

2. Open your Amazon OpenSearch Service console.

3. On the sidebar, expand **Serverless**, and then click **Dashboard**.

4. Click **Create collection**.

5. In the **Collection details** tile, for **Collection name**, enter some unique name for your new OpenSearch Serverless collection.
   Optionally, for **Description**, enter some meaningful description for your new collection.

6. For **Collection type**, select **Search**.

   <Note>
     Unstructured does not support the **Vector search** collection type. If you need vector search support, you can either continue
     with these steps to use the **Search** collection type, or you can follow the preceding steps to set up an Amazon OpenSearch Service managed cluster instead.
     However, note that the Amazon OpenSearch Serverless **Search** collection type is not as optimal as the **Vector search** collection type.
   </Note>

7. In the **Collection creation method** tile, select **Standard create**.

8. For **Encryption**, choose an AWS KMS key type.

9. For **Network access settings**, choose an **Access type**.

10. For **Resource type**, select both **Enable access to OpenSearch endpoint** and **Enable access to OpenSearch Dashboards**.

11. Click **Next**.

12. In the **Definition method** tile, select **JSON**.

13. In the **JSON editor** box, enter the following JSON, replacing the following placeholders:

    * Replace `<collection-name>` with the name of the new OpenSearch Serverless collection.
    * Replace `<account-id>` with the target AWS account ID.
    * Replace `<user-id>` with the ID of the target AWS IAM user.

    ```json theme={null}
    [
        {
            "Rules": [
                {
                    "Resource": ["collection/<collection-name>"],
                    "Permission": [
                        "aoss:CreateCollectionItems",
                        "aoss:UpdateCollectionItems",
                        "aoss:DescribeCollectionItems"
                    ],
                    "ResourceType": "collection"
                },
                {
                    "Resource": ["index/<collection-name>/*"],
                    "Permission": [
                        "aoss:CreateIndex",
                        "aoss:DescribeIndex",
                        "aoss:ReadDocument",
                        "aoss:WriteDocument",
                        "aoss:UpdateIndex",
                        "aoss:DeleteIndex"
                    ],
                    "ResourceType": "index"
                },
                {
                    "Resource": ["model/<collection-name>/*"],
                    "Permission": [
                        "aoss:DescribeMLResource",
                        "aoss:CreateMLResource",
                        "aoss:UpdateMLResource",
                        "aoss:DeleteMLResource",
                        "aoss:ExecuteMLResource"
                    ],
                    "ResourceType": "model"
                }
            ],
            "Principal": ["arn:aws:iam::<account-id>:user/<user-id>"]
        }
    ]
    ```

14. Click **Next**.

15. For **Data access policy settings**, select **Create as a new data access policy**.

16. In the **Name and description** tile, enter some unique name and an optional description for the new data access policy.

17. Click **Next**.

18. Enter any desired index details, and click **Next** again. For example:

    a. For **Index name**, enter the name of the new index in the collection.<br />
    b. For **Automatic Semantic Enrichment fields**, click **Add**, enter `embeddings` for **Automatic Semantic Enrichment field name**, click **Add**, and click **Confirm**.<br />
    c. For **Lexical search fields**, click **Add**, enter `text` for **Field name** and select **Text** for **Data type**, click **Add**, and click **Confirm**.<br />

19. Click **Submit**.

### Set up a local OpenSearch instance

The following video shows how to set up a [local OpenSearch](https://opensearch.org/downloads.html) instance.

<iframe title="YouTube video player" />

### Get the host URL

Find your host URL in the AWS console using the steps for your OpenSearch type.

#### Amazon OpenSearch Service domain

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Managed clusters**, and then click **Dashboard**.
4. In the list of available domains, click the name of your domain.
5. In the **General information** tile, copy the value of **Domain endpoint v2 (dual stack)**.

#### Amazon OpenSearch Serverless collection

1. Sign in to your AWS account, and then open your AWS Management Console.
2. Open your Amazon OpenSearch Service console.
3. On the sidebar, expand **Serverless**, and then click **Dashboard**.
4. In the list of available collections, click the name of your collection.
5. On the **Overview** tab, in the **Endpoint** tile, copy the value of **OpenSearch endpoint**.

#### Local OpenSearch instance

Your local instance URL depends on how OpenSearch is installed and configured. For guidance, see [Communicate with OpenSearch](https://opensearch.org/docs/latest/getting-started/communicate/) in the OpenSearch documentation.

### Create a search index

The name of the search index on the instance is required.

For the destination connector, if you need to create an index and you're using a master user and password as the domain's master user, you can use for example the following `curl` command. Replace the following placeholders:

* Replace `<host>` with the instance's host URL.
* Replace `<port>` with the instance's port number, which is typically `443` (for encrypted connections, and less commonly `9200` for unencrypted connections).
* Replace `<master-username>` with the master user's name, and replace `<master-password>` with the master user's password.
* Replace `<index-name>` with the name of the new search index on the instance.
* Replace `<index-schema>` with the schema for the new search index on the instance. A schema is optional; see the explanation
  following this `curl` command for more information.

```bash theme={null}
curl --request PUT "<host>:<port>/<index-name>" \
--user "<master-username>:<master-password>" \
[--header "Content-Type: application/json" \
--data '<index-schema>']
```

If you're using an existing AWS IAM user as the domain's master user instead, you should use the AWS Command Line Interface (CLI) to create the index instead of using the preceding `curl` command. To learn how, see [create-index](https://docs.aws.amazon.com/cli/latest/reference/opensearch/create-index.html) in the AWS CLI Command Reference.

For the destination connector, the index does not need to contain a schema beforehand. If Unstructured encounters an index without a schema,
Unstructured will automatically create a compatible schema for you before inserting items into the index. Nonetheless,
to reduce possible schema compatibility issues, Unstructured recommends that you create a schema that is compatible with Unstructured's schema.
Unstructured cannot provide a schema that is guaranteed to work in all
circumstances. This is because these schemas will vary based on your source files' types; how you
want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

For objects in the `metadata` field that Unstructured produces and that you want to store in an OpenSearch index, you must create fields in your index's schema that
follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
child objects:

```json theme={null}
"metadata": {
  "is_extracted": "true",
  "coordinates": {
    "points": [
      [
        134.20055555555555,
        241.36027777777795
      ],
      [
        134.20055555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        420.0269444444447
      ],
      [
        529.7005555555555,
        241.36027777777795
      ]
    ],
    "system": "PixelSpace",
    "layout_width": 1654,
    "layout_height": 2339
  },
  "filetype": "application/pdf",
  "languages": [
    "eng"
  ],
  "page_number": 1,
  "image_mime_type": "image/jpeg",
  "filename": "realestate.pdf",
  "data_source": {
    "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
    "record_locator": {
      "protocol": "file",
      "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
    }
  },
  "entities": {
    "items": [
      {
        "entity": "HOME FOR FUTURE",
        "type": "ORGANIZATION"
      },
      {
        "entity": "221 Queen Street, Melbourne VIC 3000",
        "type": "LOCATION"
      }
    ],
    "relationships": [
      {
        "from": "HOME FOR FUTURE",
        "relationship": "based_in",
        "to": "221 Queen Street, Melbourne VIC 3000"
      }
    ]
  }
}
```

You can adapt the following index schema example for your own needs. Note that outside of `metadata`, the following fields are
required by Unstructured whenever you create your own index schema:

* `element_id`
* `record_id`, which is required by Unstructured for intelligent record updates.
* `type`, which is not required, but highly recommended.
* `text`
* `embeddings` if embeddings are generated; make sure to set `dimension` to the same number of dimensions as the embedding model generates.

```json theme={null}
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "element_id": {
        "type": "keyword"
      },
      "record_id": {
        "type": "text"
      },
      "text": {
        "type": "text"
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "embeddings": {
        "type": "knn_vector",
        "dimension": 1536
      },
      "metadata": {
        "properties": {
          "is_extracted": {
            "type": "boolean"
          },
          "coordinates-points": {
            "type": "float"
          },
          "coordinates-system": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "coordinates-layout_width": {
            "type": "long"
          },
          "coordinates-layout_height": {
            "type": "long"
          },
          "filetype": {
            "type": "keyword"
          },
          "languages": {
            "type": "keyword"
          },
          "page_number": {
            "type": "integer"
          },
          "image_mime_type": {
            "type": "keyword"
          },
          "filename": {
            "type": "keyword"
          },
          "data_source-url": {
            "type": "keyword"
          },
          "data_source-record_locator-protocol": {
            "type": "keyword"
          },
          "data_source-record_locator-remote_file_path": {
            "type": "keyword"
          },
          "entities-items": {
            "properties": {
              "entity": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "entities-relationships": {
            "properties": {
              "from": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "relationship": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "to": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

See also:

* [Create an index](https://opensearch.org/docs/latest/api-reference/index-apis/create-index/)
* [Mappings and field types](https://opensearch.org/docs/latest/field-types/)
* [Explicit mapping](https://opensearch.org/docs/latest/field-types/#explicit-mapping)
* [Dynamic mapping](https://opensearch.org/docs/latest/field-types/#dynamic-mapping)
* [Unstructured document elements and metadata](/concepts/document-elements)

### Set up master user authentication

<Note>
  If you are using Enterprise Connect on a [dedicated instance](/business/dedicated-instances/overview), you do not need **master user credentials**. Skip to the **Enterprise Connect** section on this page.
</Note>

For non-local OpenSearch instances, or if you're using basic authentication to a local OpenSearch instance, you will need the master user's name and password.

For local OpenSearch instances, if you're using certificates for authentication instead of basic authentication, you will need:

* The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
* The path to the combined private key and certificate file, or
* The paths to the separate private key and certificate files.

To learn more, see [Authentication backends](https://opensearch.org/docs/latest/security/authentication-backends/authc-index/), [HTTP basic authentication](https://opensearch.org/docs/latest/security/authentication-backends/basic-authc/), and [Client certificate authentication](https://opensearch.org/docs/latest/security/authentication-backends/client-auth/).

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an OpenSearch connector to use Enterprise Connect, first set up your AWS IAM role:

### Create the IAM role

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your OpenSearch resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

### Attach a permissions policy

To allow Unstructured to access your OpenSearch installation, you may need to attach a permissions policy to the IAM role. Whether it is required depends on your OpenSearch type and how your domain access policy is configured.

#### Amazon OpenSearch Service (managed)

A permissions policy is typically not required for a managed OpenSearch service domain that has the following:

* The domain's access policy is permissive (for example, `"Principal": "*"`)
* Fine-grained access control (FGAC) is the primary access control

Both conditions above are satisfied if you selected **Easy create** in step 6 (which enables FGAC by default) and followed step 12(d) (for the domain access policy) in the [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain) section. No identity-based permissions policy is needed, and the trust policy you attached in [Create the IAM role](#create-the-iam-role) is sufficient for Enterprise Connect to function. Skip to [Grant access to OpenSearch resources](#grant-access-to-opensearch-resources).

If either condition is not met and your domain's access policy explicitly restricts access to specific principals or actions, attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:<region>:<account-id>:domain/<domain-name>/*"
    }
  ]
}
```

| Placeholder     | Value                                                  |
| --------------- | ------------------------------------------------------ |
| `<region>`      | The AWS region of the domain, for example `us-east-1`. |
| `<account-id>`  | The 12-digit ID of your AWS account.                   |
| `<domain-name>` | The name of the OpenSearch domain.                     |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

#### Amazon OpenSearch Serverless

The OpenSearch Serverless connector requires a permissions policy. Attach the following policy, replacing the placeholder values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "aoss:APIAccessAll",
      "Resource": "arn:aws:aoss:<region>:<account-id>:collection/<collection-id>"
    }
  ]
}
```

| Placeholder       | Value                                                      |
| ----------------- | ---------------------------------------------------------- |
| `<region>`        | The AWS region of the collection, for example `us-east-1`. |
| `<account-id>`    | The 12-digit ID of your AWS account.                       |
| `<collection-id>` | The ID of the OpenSearch Serverless collection.            |

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

### Grant access to OpenSearch resources

After creating the IAM role and configuring AWS permissions, you must also authorize the role within OpenSearch itself. AWS authorization and OpenSearch authorization are separate systems.

The steps in this section configure the OpenSearch authorization for your IAM role. The process you follow depends on your OpenSearch type.

#### Amazon OpenSearch Service (managed) — Map the role inside OpenSearch

You only need to map the IAM role inside OpenSearch if fine-grained access control (FGAC) is enabled on your domain:

* If you selected **Easy create** in step 6 of [Set up an AWS OpenSearch Service domain](#set-up-an-aws-opensearch-service-domain), FGAC was enabled automatically and these steps are required.
* If your domain has FGAC disabled, skip this section. Your IAM role setup is complete. Proceed to the next section to create your connector.

**Select or create an OpenSearch role**

An *OpenSearch role* is an internal security construct, distinct from an AWS IAM role. The IAM role ARN is mapped to an internal *OpenSearch role* by registering it as a **backend role**. This is how OpenSearch links an external AWS identity to an internal permission set. First, select or create the *OpenSearch role* that has permission to perform the actions your connector needs:

* For a quick proof-of-concept, use the built-in `all_access` *OpenSearch role*, already present in OpenSearch Dashboards under **Security** → **Roles**.
* For production, first [create a least-privilege *OpenSearch role*](https://opensearch.org/docs/latest/security/access-control/users-roles/) in OpenSearch Dashboards under **Security** → **Roles**, scoped to the specific indices your connector will access.

Once you have your *OpenSearch role*, complete the steps below to map your IAM role ARN to it. Skipping this mapping causes the connector to fail with a 403 error, even if the IAM role passes the AWS authorization checks:

```
security_exception ... no permissions for [indices:admin/aliases/get] and User [name=arn:aws:iam::...:role/...]
```

For more information about managing OpenSearch roles, see [Users and roles](https://opensearch.org/docs/latest/security/access-control/users-roles/) in the OpenSearch documentation.

**Map your IAM role ARN to the OpenSearch role**

You can add the mapping using OpenSearch Dashboards or the API:

* **OpenSearch Dashboards:** Go to **Security** → **Roles** → select the *OpenSearch role* → **Mapped users** tab → add the IAM role ARN under **Backend roles**. For more information, see [Fine-grained access control in Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html) in the *AWS OpenSearch Service Developer Guide*.

* Using the **API**:

  ```bash theme={null}
  curl -XPUT "$DOMAIN/_plugins/_security/api/rolesmapping/<opensearch-role-name>" \
    -H "Content-Type: application/json" \
    -u "$ADMIN_USER:$ADMIN_PASS" \
    -d '{"backend_roles":["<assumed-role-arn>"]}'
  ```

  Replace `<opensearch-role-name>` with the name of the *OpenSearch role* that you selected or created in this section. Replace `<assumed-role-arn>` with the ARN of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role).

  <Note>
    `PUT` replaces the entire role mapping. To preserve existing backend roles, first `GET` the current mapping and include all existing entries in your `PUT` request.
  </Note>

#### Amazon OpenSearch Serverless — Add the role to a data access policy

OpenSearch Serverless does not use FGAC. Authorization is managed through data access policies attached to the collection.

<Note>
  OpenSearch Serverless limits each AWS account and region to 10 data access policies. Where possible, add the IAM role ARN to an existing policy rather than creating a new one.
</Note>

Use the AWS CLI to update the collection's data access policy:

```bash theme={null}
# 1. Find the data access policy for your collection
aws opensearchserverless list-access-policies \
  --type data \
  --region <region> \
  --resource "collection/<collection-name>"

# 2. Get the current policy and note the policyVersion value in the response
aws opensearchserverless get-access-policy \
  --type data \
  --name <policy-name> \
  --region <region>

# 3. Update the policy, adding the IAM role ARN to the Principal array
aws opensearchserverless update-access-policy \
  --type data \
  --name <policy-name> \
  --policy-version <version-from-step-2> \
  --region <region> \
  --policy file://policy.json
```

The `--policy-version` value comes from the `get-access-policy` command above (#2) and enables optimistic concurrency. The `--policy` document replaces the entire policy. Include all existing principals and rules, not just your addition.

Example `policy.json`, replacing the placeholder values:

```json theme={null}
[
  {
    "Rules": [
      {
        "Resource": ["collection/<collection-name>"],
        "Permission": ["aoss:DescribeCollectionItems"],
        "ResourceType": "collection"
      },
      {
        "Resource": ["index/<collection-name>/*"],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument",
          "aoss:UpdateIndex"
        ],
        "ResourceType": "index"
      }
    ],
    "Principal": ["arn:aws:iam::<account-id>:role/<role-name>"],
    "Description": "Unstructured connector access"
  }
]
```

| Placeholder         | Value                                                                                               |
| ------------------- | --------------------------------------------------------------------------------------------------- |
| `<collection-name>` | The name of the OpenSearch Serverless collection.                                                   |
| `<account-id>`      | The 12-digit ID of your AWS account.                                                                |
| `<role-name>`       | The name of the IAM role that you created in step 2 of [Create the IAM role](#create-the-iam-role). |

**For source-only connectors:** remove `aoss:WriteDocument`,  `aoss:UpdateIndex`, and `aoss:CreateIndex` from the `Permission` array.

<Note>
  Policy changes propagate within seconds, but may occasionally take up to a minute. If the connector returns a 403 immediately after updating the policy, wait a moment and retry before treating it as a configuration error.
</Note>

## Create the source connector

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **OpenSearch**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host URL** (*required*): The OpenSearch instance's host URL, which typically takes the form of `https://<host>:<port>`.
* **Index Name** (*required*): The name of the search index on the instance.

If you're using **Basic Auth**, fill in the following fields:

* For **Authentication Method**, select **Basic Auth**.
* For **Username**, enter the master user's name.
* For **Password**, enter the master user's password.

If you're using **IAM Credentials**, fill in the following fields:

* For **Authentication Method**, select **IAM Credentials**.
* For **AWS Access Key ID**, enter the AWS access key ID for the AWS IAM user.
* For **AWS Secret Access Key**, enter the AWS secret access key for the corresponding AWS access key ID.

If you're using **IAM + Session Token**, fill in the following fields:

* For **Authentication Method**, select **IAM + Session Token**.
* For **AWS Access Key ID**, enter the temporary AWS access key ID for the AWS IAM user.
* For **AWS Secret Access Key**, enter the temporary AWS secret access key for the corresponding AWS access key ID.
* For **AWS STS Token**, enter the temporary AWS STS token for temporary access.

<Warning>
  If you are using the **IAM + Session Token** authentication method:<br /><br />
  AWS STS credentials (consisting of a temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding OpenSearch connector.<br /><br />
  You must get a new set of AWS STS credentials to replace the expired ones by, which produces a new, refreshed temporary AWS access key, temporary AWS secret access key, and temporary AWS STS session token. For more information, see [Request temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html).

  After you generate refreshed temporary AWS STS credentials, you must update the OpenSearch connector's settings with the new, refreshed AWS STS credentials.
</Warning>

If you're using **Enterprise Connect (AWS)** ([dedicated instance](/business/dedicated-instances/overview) only), fill in the following fields:

* For **Authentication Method**, select **Enterprise Connect (AWS)**.
* For **IAM Role ARN**, enter the ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
* For **External ID**, enter the unique value you chose when setting up your IAM role.


# Outlook
Source: https://docs.unstructured.io/pipelines/sources/outlook



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Outlook.

The requirements are as follows.

<iframe title="YouTube video player" />

* The Outlook user's email address.
* A Microsoft Entra ID app registration in the same Azure account as the Outlook account. You will need
  this app registration's application (client) ID, client secret, and directory (tenant) ID. [Learn how](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app).
* The Entra ID app registration must have the following Graph API permission levels of the application (not delegated) type:

  * `Mail.Read`
  * `Mail.ReadBasic`
  * `User.Read.All`

  [Learn how](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Outlook**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **User Email** (*required*): The user's email address for the target Outlook account.
* **Authentication Provider URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
* **Tenant** (*required*): The directory (tenant) ID of the Microsoft Entra ID app registration that has access to the Outlook account.
* **Client ID** (*required*): The application (client) ID of the Entra ID app registration.
* **Client Credential** (*required*): The client secret for the Entra ID app registration.
* **Folders** (*required*): A list of top-level folder names to access in the Outlook account. Separate multiple folder names by commas. Do not specify any subfolder names.
* **Recursive**: Check this box to access any and all subfolders within each of the specified top-level folders.


# Overview
Source: https://docs.unstructured.io/pipelines/sources/overview



<Tip>
  If you do not have a source location to store your documents for Unstructured to process, skip over to the
  [Dropbox source connector quickstart](/pipelines/sources/dropbox-source-quickstart). This quickstart
  guides you through the process of creating a free Dropbox account, uploading your files to Dropbox,
  and creating a source connector to connect Unstructured to those files.
</Tip>

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Source connectors are essential components in data integration systems that establish a link between your files and the data ingestion process. They facilitate the batch processing of files, allowing for the systematic retrieval and ingestion of data stored in various file formats.

<img alt="Sources in the sidebar" />

To see your existing source connectors, on the sidebar, click **Connectors**, and then click **Sources**.

To create a source connector:

1. In the sidebar, click **Connectors**.

2. Click **Sources**.

3. Click **New** or **Create Connector**.

4. For **Name**, enter some unique name for this connector.

5. In the **Provider** area, click the source location type that matches yours.

6. Click **Continue**.

7. Fill in the fields according to your connector type. To learn how, click your connector type in the following list:

   * [Azure Blob Storage](/pipelines/sources/azure-blob-storage)
   * [Box](/pipelines/sources/box)
   * [Confluence](/pipelines/sources/confluence)
   * [Couchbase](/pipelines/sources/couchbase)
   * [Databricks Volumes](/pipelines/sources/databricks-volumes)
   * [Dropbox](/pipelines/sources/dropbox)
   * [Elasticsearch](/pipelines/sources/elasticsearch)
   * [Google Cloud Storage](/pipelines/sources/google-cloud)
   * [Google Drive](/pipelines/sources/google-drive)
   * [IBM FileNet](/pipelines/sources/filenet)
   * [Jira](/pipelines/sources/jira)
   * [Kafka](/pipelines/sources/kafka)
   * [MongoDB](/pipelines/sources/mongodb)
   * [OneDrive](/pipelines/sources/onedrive)
   * [OpenSearch](/pipelines/sources/opensearch)
   * [Outlook](/pipelines/sources/outlook)
   * [PostgreSQL](/pipelines/sources/postgresql)
   * [S3](/pipelines/sources/s3)
   * [Salesforce](/pipelines/sources/salesforce)
   * [SharePoint](/pipelines/sources/sharepoint)
   * [Slack](/pipelines/sources/slack)
   * [Snowflake](/pipelines/sources/snowflake)
   * [Teradata](/pipelines/sources/teradata-sql)
   * [VAST](/pipelines/sources/vast)
   * [Zendesk](/pipelines/sources/zendesk)

8. If a **Continue** button appears, click it, and fill in any additional settings fields.

9. Click **Save and Test**.


# PostgreSQL
Source: https://docs.unstructured.io/pipelines/sources/postgresql



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from PostgreSQL.

The requirements are as follows.

* For the [Unstructured Pipelines](/pipelines/overview) or the [Unstructured API](/api-reference/overview), local PostgreSQL installations are not supported.
* For [Unstructured Ingest](/open-source/ingestion/overview), local and non-local PostgreSQL installations are supported.

The following video shows how to set up [Amazon RDS for PostgreSQL](https://aws.amazon.com/rds/postgresql/):

<iframe title="YouTube video player" />

The following video shows how to set up [Azure Database for PostgreSQL](https://azure.microsoft.com/products/postgresql):

<iframe title="YouTube video player" />

* A PostgreSQL instance.

  * [Create an Amazon RDS for PostgreSQL instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html).
  * [Create an Azure Database for PostgreSQL server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-deploy-on-azure-free-account).
  * [Install PostgreSQL locally](https://www.postgresql.org/docs/current/tutorial-install.html).

* The host name and port number for the instance.

  * For Amazon RDS for PostgreSQL, learn how to [get the host name and port number](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html#postgresql-endpoint).
  * For Azure Database for PostgreSQL, learn how to [get the host](https://learn.microsoft.com/azure/postgresql/flexible-server/quickstart-create-server#get-the-connection-information). The port number is `5432`.
  * For local PostgreSQL installations, these values are in the `postgresql.conf` file's `listen_addresses` and `port` settings. This file should be on the same machine as the instance. These values might also already be set as environment variables named `PGHOST` and `PGPORT` on the same machine as the instance.
  * For other installation types, see your PostgreSQL provider's documentation.

* Depending on your network security requirements, you might need to allow access to your instance only from specific IP addresses.

  To get Unstructured's IP address ranges, go to
  [https://assets.p6m.u10d.net/publicitems/ip-prefixes.json](https://assets.p6m.u10d.net/publicitems/ip-prefixes.json)
  and allow all of the `ip_prefix` fields' values that are listed.

  <Note>These IP address ranges are subject to change. You can always find the latest ones in the preceding file.</Note>

  To learn how to allow these IP address ranges, see your PostgreSQL provider's documentation, for example with
  [Amazon RDS for PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html) or
  [Azure Database for PostgreSQL](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-manage-firewall-portal#create-a-firewall-rule-after-server-is-created).

  <Note>
    For Amazon RDS for PostgreSQL, Amazon recommends that you set the instance's **Public access** setting to **No** by default, as this
    approach is more secure. This means that no
    resources can connect to the instance outside of the instance's associated Virtual Private Cloud (VPC) without extra configuration.
    [Learn more](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Hiding).
    [Access an Amazon RDS instance in a VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html).

    If you must enable public access, set the instance's **Public access** setting to **Yes**, and then adjust the instance's related
    security group to allow this access.
    [Learn how](https://repost.aws/en/questions/QUxemKa9u5TV6CmLiO-r5prg/lost-public-access-to-aws-rds-postgresql-instance).

    [Troubleshoot issues with connecting to Amazon RDS instances](https://repost.aws/knowledge-center/rds-connectivity-instance-subnet-vpc).
  </Note>

* A database in the instance.

  * For Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, the default database name is `postgres` unless a custom database name was specified during the instance creation process.
  * For local PostgreSQL installations, learn how to [create a database](https://www.postgresql.org/docs/current/tutorial-createdb.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* A table in the database. Learn how to [create a table](https://www.postgresql.org/docs/current/tutorial-table.html).

  For the destination connector, the table must have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the fields `id`, `element_id`, `record_id`, `text`, (and `embeddings`, if you are using `pgvector` and generating vector embeddings), as follows.
  `type` is an optional field, but highly recommended.

  If you are using `pgvector` and generating vector embeddings, the number of dimensions (in this example, `1536`) must match the number of dimensions for the associated embedding model that you use in any related Unstructured workflows or ingestion pipelines.

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT
    );
    ```

    ```sql PostgreSQL with pgvector theme={null}
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536)
    );
    ```
  </CodeGroup>

  <Info>
    The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
  </Info>

  For objects in the `metadata` field that Unstructured produces and that you want to store in PostgreSQL, you must create fields in your table's schema that
  follows Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  <CodeGroup>
    ```sql PostgreSQL theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```

    ```sql PostgreSQL with pgvector  theme={null}
    -- The fields "id", "element_id", "record_id", and "text" are required.
    -- "embeddings" is required if you are generating vector embeddings.
    --   If you are generating embeddings, the number of dimensions in "embeddings" 
    --   must match the number of dimensions for the associated embedding model 
    --   that you use in any related Unstructured workflows or ingestion pipelines.
    -- "type" is an optional field, but highly recommended.
    -- All other "metadata" fields are optional.
    CREATE EXTENSION vector;

    CREATE TABLE elements (
        id UUID PRIMARY KEY,
        element_id TEXT,
        record_id TEXT,
        text TEXT,
        type TEXT,
        embeddings vector(1536),
        is_extracted TEXT,
        points JSONB,
        system TEXT,
        layout_width INTEGER,
        layout_height INTEGER,
        filetype TEXT,
        languages TEXT[],
        page_number TEXT,
        image_mime_type TEXT,
        url TEXT,
        record_locator JSONB
    );
    ```
  </CodeGroup>

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  See also:

  * [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html) for PostgreSQL
  * [CREATE TABLE](https://github.com/pgvector/pgvector) for PostrgreSQL with pgvector
  * [Unstructured document elements and metadata](/api-reference/legacy-api/partition/document-elements)

  The following video shows how to use the `psql` utility to connect to PostgreSQL, list databases, and list and create tables:

  <iframe title="YouTube video player" />

* A user in the database, and a password for the user.

  * For Amazon RDS for PostgreSQL, learn how to [create a user](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Roles.html).
  * For Azure Database for PostgreSQL, learn how to [create a user](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-create-users).
  * For local PostgreSQL installations, learn how to [create a user](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For other installation types, see your PostgreSQL provider's documentation.

* Database access for the user.

  * For Amazon RDS for PostgreSQL, learn how to [control user access](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Access.html).
  * For Azure Database for PostgreSQL, learn how to [control user access](https://www.postgresql.org/docs/current/sql-createuser.html).
  * For local PostgreSQL installations, learn how to [give database access to a user](https://www.postgresql.org/docs/current/sql-grant.html).
  * For other installation types, see your PostgreSQL provider's documentation.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **PostgreSQL**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host** (*required*): The host name of the target PostgreSQL instance.
* **DB Name** (*required*): The name of the target database on the instance.
* **Port** (*required*): The port number on the instance.
* **Username** (*required*): The name of the target user with the appropriate access to the instance.
* **Password** (*required*): The password for the user.
* **Table Name** (*required*): The name of the target table in the database.
* **Batch Size**: The maximum number of rows to transmit at a time. The default is `100` if not otherwise specified.
* **ID Column** (*required*, source connector only): The name of the ID column in the table.
* **Columns** (*required*, source connector only): A comma separated list of column names to process (including the specified **ID Column** column). The default is all columns if not otherwise specified.

## Learn more

* <Icon icon="blog" />  [PostgreSQL Integration in the Unstructured Platform](https://unstructured.io/blog/postgresql-integration-in-the-unstructured-platform)


# S3
Source: https://docs.unstructured.io/pipelines/sources/s3



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Amazon S3.

The requirements are as follows.

The following video shows how to fulfill the minimum set of Amazon S3 requirements:

<iframe title="YouTube video player" />

<Warning>
  If you are experiencing S3 connector or workflow failures after adding a new S3 bucket or updating an existing S3 bucket,
  it could be due to S3 latency issues. You might need to wait up to a few hours before any related S3 connectors
  and workflows begin working without failures.

  Various Amazon S3 operations such as propagating DNS records for new buckets, updating bucket access policies and
  permissions, reusing bucket names after deletion, and using AWS Regions that are not geographically closer
  to your users or applications, can take a few minutes to hours to fully propagate across the Amazon network.
</Warning>

The preceding video does not show how to create an AWS account; enable anonymous access to the bucket (which is supported but
not recommended); or generate AWS STS temporary access credentials if required by your organization's security
requirements. For more information about requirements, see the following:

* An AWS account. [Create an AWS account](https://aws.amazon.com/free).

  <iframe title="YouTube video player" />

* An S3 bucket. You can create an S3 bucket by using the S3 console, following the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) or in the following video.
  Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

  <iframe title="YouTube video player" />

* Anonymous access to the bucket is supported but not recommended. (Use authenticated bucket read or write access or both instead.) To enable anonymous access, follow the steps
  [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-anonymous-user) or in the following animation.

  <img alt="Enable anonymous bucket access" />

* For authenticated bucket read or write access or both, which is recommended over anonymous access, you should first
  [block all public access to the bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/configuring-block-public-access-bucket.html).

  After blocking all public access to the bucket, for read access, the authenticated AWS IAM user must have at minimum the permissions of `s3:ListBucket` and `s3:GetObject` for that bucket.
  For write access, the authenticated AWS IAM user must have at minimum the permission of `s3:PutObject` for that bucket. Permissions
  can be granted in one of the following ways:

  * Attach the appropriate bucket policy to the bucket. See the policy examples later on this page, and [learn about bucket policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permissions remain in effect until the bucket policy is removed from the bucket.
    To apply a bucket policy by using the S3 console, follow the steps [in the S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-bucket-policy.html) or in the following video.
    Additional approaches that use AWS CloudFormation or the AWS CLI are in the how-to sections later on this page.

    <iframe title="YouTube video player" />

  * Have the IAM user temporarily assume an IAM role that contains the appropriate user policy. See the policy examples later on this page, and [learn about user policies for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html). These permission remain in effect until the assumed role's time period expires.
    Learn how to use the IAM console to [create a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html),
    [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html) that references this policy, and then
    [have the user temporarily assume the role by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
    a temporary AWS access key (`AccessKeyId`), AWS secret access key (`SecretAccessKey`), and AWS STS session token (`SessionToken`).

    <Warning>
      AWS STS credentials (consisting of an AWS access key, AWS secret access key, and AWS STS session token) can be valid for as little as 15 minutes or as long as 36 hours, depending on how the credentials were initially
      generated. After the expiry time, the credentials are no longer valid and will no longer work with the corresponding S3 connector.
      You must get a new set of credentials to replace the expired ones by [having the user temporarily assume the role again by using the AWS CLI or an AWS SDK](https://docs.aws.amazon.com/code-library/latest/ug/sts_example_sts_AssumeRole_section.html), which produces
      a new, refreshed temporary AWS access key, AWS secret access key, and AWS STS session token.

      To overwrite the expired credentials with the new set:

      * For Unstructured Pipelines, manually update the AWS Key, AWS Secret Key, and STS Token fields in Unstructured Pipelines
        for the corresponding S3 [source](/pipelines/sources/s3) or [destination](/pipelines/destinations/s3) connector.
      * For the Unstructured API, use the Unstructured API's workflow operations to call the
        [update source](/api-reference/api/source/update-source) or
        [update destination](/api-reference/api/destination/update-destination) connector operation
        for the corresponding S3 [source](/api-reference/workflow/sources/s3) or
        [destination](/api-reference/workflow/destinations/s3) connector.
      * For Unstructured Ingest, change the values of `--key`, `--secret`, and `--token` (CLI) or `key`, `secret`, and `token` (Python) in your command or code for the
        corresponding S3 [source](/open-source/ingestion/source-connectors/s3) or [destination](/open-source/ingestion/destination-connectors/s3) connector.
    </Warning>

* If you used a bucket policy intead of having the IAM user temporarily assume an IAM role for authenticated bucket access, you must provide a long-term AWS access key and secret access key for the authenticated AWS IAM user in the account.
  Create an AWS access key and secret access key by following the steps [in the IAM documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) or in the following video.

  <iframe title="YouTube video player" />

* If the target files are in the root of the bucket, you will need the path to the bucket, formatted as `protocol://bucket/` (for example, `s3://my-bucket/`).
  If the target files are in a folder, the path to the target folder in the S3 bucket, formatted as `protocol://bucket/path/to/folder/` (for example, `s3://my-bucket/my-folder/`).

* If the target files are in a folder, and authenticated bucket access is enabled, make sure the authenticated AWS IAM user has
  authenticated access to the folder as well. [See examples of authenticated folder access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-folders).

## Add an access policy to an existing bucket

To use the Amazon S3 console to add an access policy that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to an existing S3 bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

2. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/home).

3. Browse to the existing bucket and open it.

4. Click the **Permissions** tab.

5. In the **Bucket policy** area, click **Edit**.

6. In the **Policy** text area, copy the following JSON-formatted policy.
   To change the following policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this policy, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-bucket-name>` in two places with the name of your bucket.

   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::<my-account-id>:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<my-bucket-name>",
                   "arn:aws:s3:::<my-bucket-name>/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   ```

7. Click **Save changes**.

## Create a bucket with AWS CloudFormation

To use the AWS CloudFormation console to create an Amazon S3 bucket that allows all authenticated AWS IAM users
in the corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. Save the following YAML to a file on your local machine, for example `create-s3-bucket.yaml`. To change
   the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   ```yaml theme={null}
   AWSTemplateFormatVersion: '2010-09-09'
   Description: 'CloudFormation template to create an S3 bucket with specific permissions for account users.'

   Parameters:
     BucketName:
       Type: String
       Description: 'Name of the S3 bucket to create'

   Resources:
     MyS3Bucket:
       Type: 'AWS::S3::Bucket'
       Properties:
         BucketName: !Ref BucketName
         PublicAccessBlockConfiguration:
           BlockPublicAcls: true
           BlockPublicPolicy: false
           IgnorePublicAcls: true
           RestrictPublicBuckets: true

     BucketPolicy:
       Type: 'AWS::S3::BucketPolicy'
       Properties:
         Bucket: !Ref MyS3Bucket
         PolicyDocument:
           Version: '2012-10-17'
           Statement:
             - Sid: AllowAllAuthenticatedUsersInAccount
               Effect: Allow
               Principal:
                 AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
               Action:
                 - 's3:GetObject'
                 - 's3:PutObject'
                 - 's3:ListBucket'
                 - 's3:DeleteObject'
               Resource:
                 - !Sub 'arn:aws:s3:::${BucketName}'
                 - !Sub 'arn:aws:s3:::${BucketName}/*'

   Outputs:
     BucketName:
       Description: 'Name of the created S3 bucket'
       Value: !Ref MyS3Bucket
   ```

2. Sign in to the [AWS Management Console](https://console.aws.amazon.com/).

3. Open the [AWS CloudFormation Console](https://console.aws.amazon.com/cloudformation/home).

4. Click **Create stack > With new resources (standard)**.

5. On the **Create stack** page, with **Choose an existing template** already selected, select **Upload a template file**.

6. Click **Choose file**, and browse to and select the YAML file from your local machine.

7. Click **Next**.

8. Enter a unique **Stack name** and **BucketName**.

9. Click **Next** two times.

10. Click **Submit**.

11. Wait until the **Status** changes to **CREATE\_COMPLETE**.

12. After the bucket is created, you can delete the YAML file, if you want.

## Create a bucket with the AWS CLI

To use the AWS CLI to create an Amazon S3 bucket that allows all authenticated AWS IAM users in the
corresponding AWS account to read and write to the bucket, do the following.

<Info>Your organization might have stricter bucket policy requirements. Check with your AWS account
administrator if you are unsure.</Info>

1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).

2. [Set up the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html).

3. Copy the following script to a file on your local machine, for example a file named `create-s3-bucket.sh`.
   To change the following bucket policy to restrict it to a specific user in the AWS account, change `root` to that
   specific username.

   In this script, replace the following:

   * Replace `<my-account-id>` with your AWS account ID.
   * Replace `<my-unique-bucket-name>` with the name of your bucket.
   * Replace `<us-east-1>` with your AWS Region.

   ```bash theme={null}
   #!/bin/bash

   # Set variables for the AWS account ID, Amazon S3 bucket name, and AWS Region.
   ACCOUNT_ID="<my-account-id>"
   BUCKET_NAME="<my-unique-bucket-name>"
   REGION="<us-east-1>"

   # Temporary filename for the bucket policy.
   # Do not change this variable.
   POLICY_FILE="bucket_policy.json"

   # Create the bucket.
   aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION

   # Wait for the bucket to exist.
   echo "Waiting for bucket '$BUCKET_NAME' to be fully created..."
   aws s3api wait bucket-exists --bucket $BUCKET_NAME

   # Check if the wait command was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket '$BUCKET_NAME' has been fully created."
   else
       echo "Error: Timed out waiting for bucket '$BUCKET_NAME' to be created."
       exit 1
   fi

   # Remove the "block public policy" bucket access setting.
   aws s3api put-public-access-block \
       --bucket $BUCKET_NAME \
       --public-access-block-configuration \
       '{"BlockPublicPolicy": false, "IgnorePublicAcls": false, "BlockPublicAcls": false, "RestrictPublicBuckets": false}'

   # Check if the operation was successful.
   if [ $? -eq 0 ]; then
       echo "The block public policy access setting was removed from '$BUCKET_NAME'."
   else
       echo "Error: Failed to remove the block public policy access setting from '$BUCKET_NAME'."
       exit 1
   fi

   # Create the bucket policy.
   cat << EOF > $POLICY_FILE
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowAuthenticatedUsersInAccountReadWrite",
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::$ACCOUNT_ID:root"
               },
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::$BUCKET_NAME",
                   "arn:aws:s3:::$BUCKET_NAME/*"
               ],
               "Condition": {
                   "StringEquals": {
                       "aws:PrincipalType": "IAMUser"
                   }
               }
           }
       ]
   }
   EOF

   # Apply the bucket policy.
   aws s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://$POLICY_FILE

   # Check if the policy application was successful.
   if [ $? -eq 0 ]; then
       echo "The bucket policy was applied to '$BUCKET_NAME'."
   else
       echo "Error: Failed to apply the bucket policy to '$BUCKET_NAME'."
       exit 1
   fi

   # Verify the applied policy.
   echo "Verifying the applied policy:"
   aws s3api get-bucket-policy --bucket $BUCKET_NAME --query Policy --output text

   # Remove the temporary bucket policy file.
   rm $POLICY_FILE
   ```

4. Run the script, for example:

   ```bash theme={null}
   sh create-s3-bucket.sh
   ```

5. After the bucket is created, you can delete the script file, if you want.

## Work with user-defined metadata

User-defined metadata in S3 is metadata that you can choose to set at the time that you upload a file to an S3 bucket.
User-defined metadata is specified in S3 as a set of name-value pairs. Each name-value pair begins with `x-amz-meta-` and is
followed by a unique name.

For more information about how to add or replace user-defined metadata for a file in S3, see the following:

* [Working with object metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html)
* [User-defined metadata](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata)
* [Editing object metadata in the Amazon S3 console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/add-object-metadata.html)

Unstructured outputs any user-defined metadata that it finds for a file into the `metadata.data_source.record_locator.metadata` field of
the document elements' output for the corresponding file. For example, if Unstructured processes a file with the user-defined metadata
`x-amz-meta-mymetadata` name set to the value `myvalue`, Unstructured outputs the following into the `metadata.data_source.record_locator.metadata` field of
the document elements' output for the corresponding file:

```json theme={null}
[
    {
        "type": "...",
        "element_id": "...",
        "text": "...",
        "metadata": {
            "data_source": {
                "record_locator": {
                    "metadata": {
                        "mymetadata": "myvalue"
                    }
                }
            }
        }
    }
]
```

## FIPS and ambient credentials

<Note>
  The following information applies to [Unstructured Business dedicated instance and in-VPC](/business/overview) deployments only.
</Note>

Unstructured Business accounts support the Federal Information Processing Standard (FIPS) for
Amazon S3. [Learn more about AWS support for FIPS](https://aws.amazon.com/compliance/fips/). Specifically,
when creating an S3 connector with the [Unstructured Pipelines](/pipelines/overview) or
[Unstructured API's workflow operations](/api-reference/workflow/overview), Unstructured Business dedicated instance and in-VPC deployments
support the use of `fips`-based
[S3 endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html) and, for authenticated access to
S3 FIPS buckets, S3 ambient credentials.

To use Unstructured Pipelines to set up an S3 source or destination connector
to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.
2. When creating the connector, for the S3 connector's **Bucket URI** field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).
3. For the **Authentication Method** field, select **Ambient Credentials**.
4. Check the box labeled **Use Ambient Credentials**.
5. Save and test the connector.

To use the Unstructured API's workflow operations to set up an S3 source or
destination connector to use an S3 FIPS bucket and S3 ambient credentials, do the following:

1. Create an environment variable named `ALLOW_AMBIENT_CREDENTIALS_S3`, and set its value to `true`.

2. When creating the connector,
   for the `config` parameter's `remote_url` field, specify the path to the S3 FIPS bucket, formatted as
   `https://<bucket-name>.<endpoint>`, for example
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com`.
   If the target files are in a folder, specify the path to the target folder in the S3 FIPS bucket instead,
   formatted as `https://<bucket-name>.<endpoint>/path/to/folder` (for example,
   `https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/`).

3. For the `config` parameter, add an `ambient_credentials` field, and set its value to true. For example:

   <CodeGroup>
     ```python Python SDK theme={null}
     # ...
     config={
         "...": "...",
         "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
         "ambient_credentials": True
     }
     # ...
     ```

     ```bash curl theme={null}
     # ...
     {
         "...": "...", 
         "config": {
             "...": "...",
             "remote_url": "https://my-bucket-name.s3-fips.us-gov-east-1.amazonaws.com/my-folder/",
             "ambient_credentials": "true"
         }
     }
     # ...
     ```
   </CodeGroup>

4. Run your code to create the connector.

## Set up Enterprise Connect authentication

<Note>
  Enterprise Connect is available for [dedicated instance](/business/dedicated-instances/overview) customers only, and must be enabled on your instance before use. Contact your Unstructured account team or [Unstructured Support](https://support.unstructured.io/) to request access and have it enabled.
</Note>

Enterprise Connect is an authentication method for AWS connectors. During a workflow run, Unstructured assumes an IAM role in your AWS account and uses short-lived credentials scoped to that operation. Credentials are never stored and expire automatically. For an overview, see [Enterprise Connect for AWS](/business/aws/enterprise-connect).

To configure an S3 connector to use Enterprise Connect, first set up your AWS IAM role:

1. Choose an **External ID** — a unique value that prevents unauthorized parties from assuming your IAM role. You will add this value to your AWS trust policy and enter it in the Unstructured connector. Use upper and lower case alphanumeric characters, underscores, or any of `+=,.@:\/-`; no spaces; 2–1224 characters.

2. In your AWS account, create an IAM role that Unstructured will assume to access your S3 resources (for example, `unstructured-connector-role`), or use an existing one. For more information, see [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) in the *AWS IAM User Guide*.

   Attach the following trust policy, replacing the placeholder values:

   ```json theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "<unstructured-service-role-arn>"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "<external-id>"
           }
         }
       }
     ]
   }
   ```

   | Placeholder                       | Value                                                                                                                     |
   | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
   | `<unstructured-service-role-arn>` | The ARN of the Unstructured service role for your dedicated instance. Get this value from your Unstructured account team. |
   | `<external-id>`                   | The unique value you chose as your External ID.                                                                           |

   The `sts:ExternalId` condition prevents the [confused deputy problem](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). It ensures only your Unstructured workspace can use this role, even if another party knows the Unstructured service role ARN.

3. Attach a permissions policy to the role that grants access to your S3 bucket:

   * **Source connector:** at minimum `s3:GetObject`, `s3:ListBucket`
   * **Destination connector:** at minimum `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`

   Scope the policy to the specific bucket or prefix your connector will access.

<Tip>
  You can attach the permissions policy immediately after creating the role, without leaving the AWS IAM console. Continue following the [Create a role using custom trust policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html) guide in the *AWS IAM User Guide* referenced in the previous step.
</Tip>

Then, create the connector by following the steps in the **Create the  connector** section on this page.

## Create the source connector

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Amazon S3**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Bucket URI** (*required*): The URI for the bucket or folder, formatted as `s3://my-bucket/` (if the files are in the bucket's root) or `s3://my-bucket/my-folder/`.
* **Recursive** (source connector only): Check this box to access subfolders within the bucket.

If you're using **Anonymous**, fill in the following field:

* For **Authentication Method**, select **Anonymous**.

If you're using **Secret**, fill in the following fields:

* For **Authentication Method**, select **Secret**.
* For **AWS Key**, enter the AWS access key ID for the authenticated AWS IAM user.
* For **AWS Secret Key**, enter the AWS secret access key corresponding to the preceding AWS access key ID.

If you're using **Enterprise Connect (AWS)** ([dedicated instance](/business/dedicated-instances/overview) only), fill in the following fields:

* For **Authentication Method**, select **Enterprise Connect (AWS)**.
* For **IAM Role ARN**, enter the ARN of the IAM role Unstructured will assume via AWS STS. For example, `arn:aws:iam::123456789012:role/MyRole`.
* For **External ID**, enter the unique value you chose when setting up your IAM role.

For all authentication methods, you can also fill in the following field:

* For **Custom URL**, enter a custom endpoint URL if connecting to a non-AWS S3 bucket.


# Salesforce
Source: https://docs.unstructured.io/pipelines/sources/salesforce



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Salesforce.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Salesforce account](https://developer.salesforce.com/signup).

* Your Salesforce username. To get this username, do the following:

  1. Log in to your Salesforce account.
  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.
  3. In the **Home** tab, under **Administration**, expand **Users**, and then click **Users**.
  4. Note the **User Name** value (*not* the **Name** value) for your user.

* The names of the Salesforce categories (objects) that you want to access, specified as a comma-separated list.
  Available categories include `Account`, `Campaign`, `Case`, `EmailMessage`, and `Lead`.

* A Salesforce connected app in your Salesforce account.

  If you do not already have a Salesforce connected app, to create one, start by creating or getting the certificate (`.crt`) and private key (`.pem`) files
  that you will associate with the connected app.

  If you do not have the certificate and private key files, you can use the `openssl` utility on your local machine to create your own
  private key and self-signed certificate, if your organization allows, by running the following commands, one command at a time:

  ```bash theme={null}
  openssl genrsa -out MyPrivateKey.pem -traditional
  openssl req -new -key MyPrivateKey.pem -out MyCertificateSigningRequest.csr
  openssl x509 -req -in MyCertificateSigningRequest.csr -signkey MyPrivateKey.pem -out MyCertificate.crt -days 365
  ```

  Of course, you can change these preceding example filenames as needed. Be sure to store these generated files in a secure location.

  To create a Salesforce connected app, do the following:

  1. Log in to your Salesforce account.

  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.

  3. In the **Home** tab, under **Platform Tools**, expand **Apps**, and then click **App Manager**.

  4. Click **New Connected App**.

  5. With **Create a Connected App** selected, click **Continue**.

  6. At a minimum, fill in the following, and then click **Save**:

     * **Connected App Name**

     * **API Name** (can be the same as **Connected App Name**, but do not use spaces or punctuation)

     * **Contact Email**

     * Under **API (Enable OAuth Settings)**, check **Enable OAuth Settings**.

     * For **Callback URL**, entering `https://localhost` is okay if you won't be using this connected app for other special authentication scenarios.

     * Check **Use digital signatures**, click **Choose File**, and browse to and select your certificate (`.crt`) file.

     * For **Selected OAuth Scopes**, move the following entries from the **Available OAuth Scopes** list to the **Selected OAuth Scopes** list:

       * **Manage user data via APIs (api)**
       * **Perform requests on your behalf at any time (refresh\_token, offline\_access)**

     * Uncheck **Require Proof Key for Code Exchange (PKCE) Extension for Supported Authorization Flows**.

     * Leave **Require Secret for Web Server Flow** checked.

     * Leave **Require Secret for Refresh Token Flow** checked.

     * Check **Enable Authorization Code and Credentials Flow**.

  7. On the connected app's details page, click **Manage**, click **Edit Policies**, set the following under **OAuth Policies**, and then click **Save**:

     * Set **Permitted Users** to **All users may self-authorize**.
     * Set **IP Relaxation** to **Relax IP restrictions**.
     * Set **Refresh Token Policy** to **Refresh token is valid until revoked**.

* The OAuth consumer key (client ID) for the Salesforce connected app.

  To get the Salesforce connected app's consumer key, do the following:

  1. Log in to your Salesforce account.
  2. In the top navigation bar, click the **Quick Settings** (gear) icon, and then click **Open Advanced Setup**.
  3. In the **Home** tab, under **Platform Tools**, expand **Apps**, and then click **App Manager**.
  4. In the list of apps, click the arrow next to the target connected app, and click **View**.
  5. Click **Manage Consumer Details**.
  6. Complete the on-screen security verification.
  7. Note the **Consumer Key** value.

* You must use your Salesforce account to do a one-time approval of the Salesforce connected app by using its consumer key and callback URL. To do this, while you are logged in to your
  Salesforce account, browse to the following URL, replacing `<client-id>` with the consumer key value. This URL assumes that the callback URL
  is `https://localhost`:

  ```
  https://login.salesforce.com/services/oauth2/authorize?response_type=code&client_id=<client-id>&redirect_uri=https%3A%2F%2Flocalhost
  ```

* To ensure maximum compatibility across Unstructured service offerings, you should give the contents of the private key (`.pem`) file to Unstructured as
  a string that contains the contents of the file (*not* the private key file itself).\
  To print this string suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
  In this command, replace `<path-to-private-key-file>` with the path to the private key file.

  * For macOS or Linux:

    ```bash theme={null}
    cat <path-to-private-key-file>
    ```

  * For Windows:

    ```text theme={null}
    Get-Content <path-to-private-key-file>
    ```

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Salesforce**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Username** (*required*): The Salesforce username that has access to the target Salesforce categories (objects).
* **Salesforce categories**: A comma-separated list of the Salesforce categories to access. Available categories include
  `Account`, `Campaign`, `Case`, `EmailMessage`, and `Lead`.
* **Consumer key** (*required*): The consumer key (client ID) for the target Salesforce connected app.
* **Private key (PEM)** (*required*): The private key (PEM) associated with the consumer key for the Salesforce connected app. The PEM must be expressed as a single-line string.


# SharePoint
Source: https://docs.unstructured.io/pipelines/sources/sharepoint



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from SharePoint.

The requirements are as follows.

<Note>
  If you are setting up the SharePoint connector for the first time, you can skip past this note.

  Previous versions of the SharePoint connector relied on SharePoint app principals for authentication. Current versions of the
  SharePoint connector no longer support these SharePoint app principals. Microsoft deprecated support for Share Point app principals on November 27, 2023.
  SharePoint app principals will no longer work for SharePoint tenants that were created on or after November 1, 2024, and they will stop working
  for all SharePoint tenants as of April 2, 2026. [Learn more](https://learn.microsoft.com/sharepoint/dev/sp-add-ins/retirement-announcement-for-azure-acs).

  Current versions of the SharePoint connector now rely on Microsoft Entra ID app registrations for authentication.

  To migrate from SharePoint app princpals to Entra ID app regisrations, replace the following settings in your existing SharePoint connector,
  as listed in the requirements following this note:

  * Replace the deprecated SharePoint app principal's application client ID value with your replacement Entra ID app registration's **Application (client) ID** value.
  * Replace the deprecated SharePoint app principal's client secret value with your replacement Entra ID app registration's **Client secret** value.
  * Add your replacement Entra ID app registration's **Directory (tenant) ID** value, token authority URL value, and the correct set of Microsoft Graph access permissions for SharePoint Online.

  If you need migration help, get assistance from our [Slack community](https://short.unstructured.io/pzw05l7), or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
</Note>

* A SharePoint Online plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes SharePoint Online.
  [Learn more](https://www.microsoft.com/en-us/microsoft-365/SharePoint/compare-SharePoint-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).

* A OneDrive for business plan, or a Microsoft 365 or Office 365 Business or enterprise plan that includes OneDrive.
  (Even if you only plan to use SharePoint Online, you still need a plan that includes OneDrive, because the SharePoint connector is built on OneDrive technology.)
  [Learn more](https://www.microsoft.com/microsoft-365/onedrive/compare-onedrive-plans).
  [Shop for business plans](https://www.microsoft.com/microsoft-365/business/compare-all-microsoft-365-business-products).
  [Shop for enterprise plans](https://www.microsoft.com/microsoft-365/enterprise/microsoft365-plans-and-pricing).
  OneDrive personal accounts, and Microsoft 365 Free, Basic, Personal, and Family plans are not supported.

* The SharePoint Online and OneDrive plans must share the same Microsoft Entra ID tenant.
  [Learn more](https://learn.microsoft.com/microsoft-365/enterprise/subscriptions-licenses-accounts-and-tenants-for-microsoft-cloud-offerings?view=o365-worldwide).

* The SharePoint Online site base URL. This URL must meet the following requirements:

  1. Starts with `https://`.
  2. Followed by any number of uppercase or lowercase letters, digits, or hyphens (`-`). This portion of the URL must contain at least
     one dot segment after the first domain label; for example, `.com`, `.sharepoint.com`, or `.co.uk`.
  3. Followed optionally by `/sites/` or `/teams/` and ends with any name that can include any number of
     uppercase or lowercase letters, digits, hyphens (`-`), or underscores (`_`); for example, `/sites/my-site` or `/teams/my_team`.

  Some examples of valid site URLs include:

  * `https://contoso.com`
  * `https://my-company.sharepoint.com`
  * `https://example.co.uk`
  * `https://contoso.sharepoint.com/sites/engineering`
  * `https://teams.example.com/teams/marketing`

  Additionally, note the following about collection-level URLs and all site URLs:

  * Site collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com/sites/<site-collection-name>`.
  * Team collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com/teams/<team-collection-name>`.
  * Root site collection-level URLs typically have a format with a pattern similar to `https://<tenant>.sharepoint.com`.
  * To process all sites within a SharePoint tenant, use a site URL with a pattern similar to `https://<tenant>-admin.sharepoint.com`.

  [Learn more](https://learn.microsoft.com/microsoft-365/community/query-string-url-tricks-sharepoint-m365).

* The display name of the SharePoint Online library to use. The default is `Documents`.

* The path to the SharePoint Online library to use. By default, the root of the target library is used.
  To start from a path other than the root, enter the path that you want to use, beginning from the root. For example, to use
  the **my-folder > my-subfolder** path in the target library, you would specify `my-folder/my-subfolder`.

  The following video shows how to get the site URL and a path within the site:

  <iframe title="YouTube video player" />

* Two types of authentication are supported: client credentials and a username and password. Both authentication types require a
  Microsoft Entra ID app registration. You will need to provide
  the **Application (client) ID**, **Directory (tenant) ID**, and **Client secret** for the Entra ID app registration, and the
  app registration must have the correct set of Microsoft Graph access permissions. These permissions include:

  * `Sites.ReadWrite.All` (if both reading and writing are needed)
  * `User.Read.All`
    [Learn more](https://learn.microsoft.com/answers/questions/2116616/service-principal-access-to-sharepoint-online).

  1. [Create an Entra ID app registration](https://learn.microsoft.com/entra/identity-platform/quickstart-register-app?pivots=portal).
  2. [Add Graph access permissions to an app registration](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#add-permissions-to-an-application).
  3. [Grant consent for the added Graph permissions](https://learn.microsoft.com/entra/identity-platform/howto-update-permissions?pivots=portal#grant-consent-for-the-added-permissions-for-the-enterprise-application).

  The following video shows how to create an Entra ID app registration:

  <iframe title="YouTube video player" />

  The following video shows how to add the correct set of Graph access permissions to the Entra ID app registration:

  <iframe title="YouTube video player" />

* The token authority URL for your Microsoft Entra ID app registration. This is typically `https://login.microsoftonline.com`

* For username and password authentication, you must also provide the User Principal Name (UPN) and its password for the OneDrive account in the Microsoft Entra ID tenant. This UPN is typically the OneDrive account user's email address. To find a UPN:

  1. Depending on your plan, sign in to your Microsoft 365 admin center (typically [https://admin.microsoft.com](https://admin.microsoft.com)) using your administrator credentials,
     or sign in to your Office 365 portal (typically [https://portal.office.com](https://portal.office.com)) using your credentials.
  2. In the **Users** section, click **Active users**.
  3. Locate the user account in the list of active users.
  4. The UPN is displayed in the **Username** column.

  The following video shows how to get a UPN:

  <iframe title="YouTube video player" />

## Document permissions metadata

The source connector outputs any permissions information that it can find in the source location about the processed source documents and associates that information with each
corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field. This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.

<Warning>
  The permissions metadata Unstructured outputs should not be used for runtime authorization
  or access control enforcement.

  Unstructured outputs document permissions metadata that is accurate only
  at the point in time when Unstructured ingested the corresponding document
  to which those permissions applied. Because this metadata is a
  point-in-time copy of the permissions in the source location, these
  metadata outputs that are sent to your destination location are not
  always guaranteed to match the current permissions in the source location.

  Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.

  This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
  *permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
  workflow (in other words, if **Reprocess All Files** is turned off or set
  to false for a workflow), that worfklow will not output metadata for any
  document permissions that have been added, changed, or removed since the
  previous workflow run, *unless* the corresponding documents' content
  has also been changed since the previous workflow run.
</Warning>

The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.

```json theme={null}
[
    {
        "...": "...",
        "metadata": {
            "...": "...",
            "data_source": {
                "...": "...",
                "permissions_data": [
                    {
                        "read": {
                            "users": [
                                "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
                            ],
                            "groups": [
                                "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
                                "cccccccc-cccc-cccc-cccc-cccccccccccc"
                            ]
                        }
                    },
                    {
                        "update": {
                            "users": [
                                "dddddddd-dddd-dddd-dddd-dddddddddddd",
                                "eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee"
                            ],
                            "groups": [
                                "ffffffff-ffff-ffff-ffff-ffffffffffff"
                            ]
                        }
                    },
                    {
                        "delete": {
                            "users": [
                                "11111111-1111-1111-1111-111111111111"
                            ],
                            "groups": [
                                "22222222-2222-2222-2222-222222222222"
                            ]
                        }
                    }
                ],
                "...": "..."
            }
        }
    }
]
```

To look up information about a particular Microsoft Entra ID user, use the user's ID along with the
[GET /users](https://learn.microsoft.com/graph/api/user-get) operation in the Microsoft Graph API.

To look up information about a particular Microsoft Entra ID group, use the group's ID along with the
[GET /groups](https://learn.microsoft.com/graph/api/group-get) operation in the Microsoft Graph API.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **SharePoint**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Site URL** (*required*): The base URL of the SharePoint site to connect to.
* **Documents Library**: The display name of the SharePoint library to use. The default is `Documents`.
* **Path**: The path to use within the library. The default is the root of the target library. To use a different path, specify the correct path format as described previously in this article.
* **Recursive**: Check this box to recursively process data from subfolders within the target path.
* **Client ID** (*required*): The **Application (client) ID** for the Microsoft Entra ID app registration with the correct set of Microsoft Graph access permissions.
* **Tenant ID** (*required*): The **Directory (tenant) ID** for the Entra ID app registration.
* **Authority URL** (*required*): The authentication token provider URL for the Entra ID app registration. The default is `https://login.microsoftonline.com`.
* **Client Credentials** (*required*): The **Client secret** for the Entra ID app registration.
* **User Principal Name (UPN)** (*required* for username and password authentication): For username and password authentication, the UPN for the OneDrive account in the Entra ID tenant.
* **Password** (*required* for username and password authentication): For username and password authentication, the password for the UPN.


# Slack
Source: https://docs.unstructured.io/pipelines/sources/slack



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Slack.

The requirements are as follows.

<iframe title="YouTube video player" />

* A Slack app. Create a Slack app by following [Step 1: Creating an app](https://api.slack.com/quickstart#creating).

* The app must have the following OAuth scopes:

  * To read messages from public channels, add the scopes `channels:join` and `channels:history`.
  * To read messages from private channels, add the scope `groups:history`.

  Add these scopes to the app by following [Step 2: Requesting scopes](https://api.slack.com/quickstart#scopes).

* The app must be installed and authorized for the target Slack workspace. Install and authorize the app by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).

* The app's access token. Get this token by following [Step 3: Installing and authorizing the app](https://api.slack.com/quickstart#installing).

* Add the app to the target channels in the Slack workspace. To do this from the channel, open the channel's details page, click the **Integrations** tab, click **Add apps**, and follow the on-screen directions to install the app.

* The channel ID for each target channel. To get this ID, open the channel's details page, and look for the **Channel ID** field on the **About** tab.

* The starting and ending date and time range for the channels to be processed. Supported formats include:

  * `YYYY-MM-DD`
  * `YYYY-MM-DDTHH:MM:SS`
  * `YYYY-MM-DDTHH:MM:SSZ`
  * `YYYY-MM-DD+HH:MM:SS`
  * `YYYY-MM-DD-HH:MM:SS`

<Info>
  Slack has established API rate limits that could restrict the number and frequency of messages that can be read. To
  explore possible workarounds for these cases, see your Slack representative.
</Info>

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Click **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Slack**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Channels** (*required*): A comma-separated list of the target Slack channel IDs from which to read messages.
* **Start Date**: The date and time, in ISO 8601 format (such as `YYYY-MM-DDTHH:MM:SSZ`), from which to begin reading messages. Messages created before this timestamp are ignored. By default, all messages are read unless an end date is specified.
* **End Date**: The date and time, in ISO 8601 format (such as `YYYY-MM-DDTHH:MM:SSZ`), at which to stop reading messages. Messages created after this timestamp are ignored. By default, all messages are read unless a start date is specified.
* **Bot Token** (*required*): The access token for the Slack app that has permission to read messages from the target Slack channels.


# Snowflake
Source: https://docs.unstructured.io/pipelines/sources/snowflake



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Snowflake.

The requirements are as follows.

* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.

  <iframe title="YouTube video player" />

  To get the identifier for the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
  3. On the **Account** tab, note the value of the **Account Identifier** field.

  Alternatively, the following Snowflake query returns the current account's identifier:

  ```text theme={null}
  SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
  ```

* A Snowflake user, which can be a service user (recommended) or a human user.

  To create a service user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Projects > Worksheets**.

  3. Click the **+** button to create a SQL worksheet.

  4. In the worksheet, enter the following Snowflake query to create a service user, replacing the following placeholders:

     * Replace `<service-user-name>` with some name for the service user.
     * Replace `<default-role-name>` with the name of any default role for the service user to use.

     ```sql theme={null}
     CREATE USER <service-user-name>
     DEFAULT_ROLE = "<default-role-name>"
     TYPE = SERVICE
     ```

  5. Click the arrow icon to run the worksheet, which creates the service user.

  6. To get their login name, on the navigation menu, click **Admin > Users & Roles**.

  7. On the **Users** tab, in the list of available users, click the name of the target user.

  8. In the **About** tile, note the **Login Name** for the user.

  To create a human user entry and get their login name (not username):

  <iframe title="YouTube video player" />

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & roles**.
  3. Click the **Users** tab.
  4. Click **+ User**.
  5. Follow the on-screen guidance to specify the user's settings.
  6. Click **Create User**.
  7. To get their login name, on the navigation menu, click **Admin > Users & Roles**.
  8. On the **Users** tab, in the list of available users, click the name of the target user.
  9. In the **About** tile, note the **Login Name** for the user.

* A programmatic access token (PAT) for the Snowflake user.

  <iframe title="YouTube video player" />

  To create a programmatic access token (PAT) for a user:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.

  3. On the **Users** tab, in the list of available users, click the name of the target user.

  4. In the **Programmatic access tokens** tile, click the **Generate new token** button.

  5. Follow the on-screen guidance to specify the PAT's settings.

     <Warning>
       You must set an expiration date for the PAT. This expiration date can be as soon as one day after the PAT is created or up to one year or even later.
       Once this PAT expires, the connector will stop working.
       To make sure that your connector continues to work, before your current PAT expires, you must follow this procedure again to generate a new PAT and
       update your connector's settings with your new PAT's value.

       Unstructured does not notify you when a PAT is about to expire or has already expired.
       You are responsible for tracking your PATs' expiration dates and taking corrective action before they expire.
     </Warning>

  6. Click **Generate**.

  7. Copy the generated PAT's value to a secure location, as you will not be able to access it again. If you lose this PAT's value,
     you will need to repeat this procedure to generate a new, replacement one.

  The PAT will not work unless the Snowflake account also has a valid
  [network rule](https://docs.snowflake.com/en/user-guide/network-rules) along with a valid
  [network policy](https://docs.snowflake.com/en/user-guide/network-policies) attached to that rule.
  The network rule must also be activated on the Snowflake account to begin taking effect.

  To create a valid network rule:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.

  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Rules**.

  3. Click **+ Network Rule**.

  4. Enter some name for the network rule.

  5. For **Type**, select **IPv4**.

  6. For **Mode**, select **Ingress**.

  7. For **Identifiers**, next to the magnifying glass icon, enter `0.0.0.0/0`, and then press **Enter**.

     <Note>
       The `0.0.0.0/0` value allows all IP addresses to access the Snowflake account.
       You can specify a more specific IP address range if you prefer. However, this more specific IP address range
       will apply to all users, including the user for which you created the PAT.
     </Note>

  8. Click **Create Network Rule**.

  To create a valid network policy, attaching the preceding network rule to this policy at the same time:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click **+ Network Policy**.
  4. Enter some name for the network policy.
  5. Make sure **Allowed** is selected.
  6. In the **Select rule** drop-down list, select the precedingnetwork rule to attach to this network policy.
  7. Click **Create Network Policy**.

  To activate the network rule in the account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Security > Network Policies**.
  3. Click the name of the precedingnetwork policy to activate.
  4. In the policy's side panel, click the ellipsis (three dots) icon, and then click **Activate On Account**.
  5. Click **Activate policy**.

* (No longer recommended, as passwords are being deprecated by Snowflake—use PATs instead) The Snowflake [user's login name (not username) and the user's password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
  This user must be a human user. Passwords are not supported for service users.

* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.

  <iframe title="YouTube video player" />

  * To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
  * To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
  * To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
  * To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
  * To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.

  To view a list of available roles in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
  3. Click the **Roles** tab.

  Alternatively, the following Snowflake query returns a list of available roles in the current account:

  ```text theme={null}
  SHOW ROLES;
  ```

  [Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).

* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.

  <iframe title="YouTube video player" />

  To view a list of available warehouses in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.

  The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:

  ```text theme={null}
  SELECT t.VALUE:type::VARCHAR as type,
         t.VALUE:host::VARCHAR as host,
         t.VALUE:port as port
  FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
  ```

* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.

  <iframe title="YouTube video player" />

  To view a list of available databases in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.

  Alternatively, the following Snowflake query returns a list of available databases in the current account:

  ```text theme={null}
  SHOW DATABASES;
  ```

* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.

  To view a list of available schemas for a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the target database.

  Alternatively, the following Snowflake query returns a list of available schemas in the current account:

  ```text theme={null}
  SHOW SCHEMAS;
  ```

  The following Snowflake query returns a list of available schemas for the database named `<database_name>` in the current account:

  ```text theme={null}
  SHOW SCHEMAS IN DATABASE <database_name>;
  ```

* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.

  To view a list of available tables for a schema in a database in the current Snowflake account:

  1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
  2. In Snowsight, on the navigation menu, click **Data > Databases**.
  3. Expand the name of the database that contains the target schema.
  4. Expand the name of the target schema.
  5. Expand **Tables**.

  Alternatively, the following Snowflake query returns a list of available tables for the schema named `<schema_name>` in the datbase named
  `<database_name>` in the current account:

  ```text theme={null}
  SHOW TABLES IN SCHEMA <database_name>.<schema_name>;
  ```

  Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The minimum viable
  schema for Unstructured contains only the columns `ID`, `ELEMENT_ID`, and `RECORD_ID`. The columns `TEXT` and `TYPE` are optional, but highly recommended.
  If you are generating embeddings, then you must also include the column `EMBEDDINGS`.

  In the following `CREATE TABLE` statement, replace the following placeholders with the appropriate values:

  * `<database_name>`: The name of the target database in the Snowflake account.
  * `<schema_name>`: The name of the target schema in the database.
  * `<number-of-dimensions>`: The number of dimensions for any embeddings that you plan to use. This value must match the number of dimensions for any embeddings that are
    specified in your related Unstructured workflows or pipelines. If you plan to use Snowflake vector embedding generation or Snowflake vector search,
    this value must match the number of dimensions that you plan to have Snowflake generate or search against.

  ```sql SQL theme={null}
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      ELEMENT_ID VARCHAR,
      RECORD_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>)
  );
  ```

  For objects in the `metadata` field that Unstructured produces and that you want to store in a Snowflake table, you must create columns in your table's schema that
  follow Unstructured's `metadata` field naming convention. For example, if Unstructured produces a `metadata` field with the following
  child objects:

  ```json theme={null}
  "metadata": {
    "is_extracted": "true",
    "coordinates": {
      "points": [
        [
          134.20055555555555,
          241.36027777777795
        ],
        [
          134.20055555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          420.0269444444447
        ],
        [
          529.7005555555555,
          241.36027777777795
        ]
      ],
      "system": "PixelSpace",
      "layout_width": 1654,
      "layout_height": 2339
    },
    "filetype": "application/pdf",
    "languages": [
      "eng"
    ],
    "page_number": 1,
    "image_mime_type": "image/jpeg",
    "filename": "realestate.pdf",
    "data_source": {
      "url": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf",
      "record_locator": {
        "protocol": "file",
        "remote_file_path": "file:///home/etl/node/downloads/00000000-0000-0000-0000-000000000001/7458635f-realestate.pdf"
      }
    }
  }
  ```

  You could create corresponding fields in your table's schema by using the following field names and data types:

  ```sql theme={null}
  -- The ID, RECORD_ID, and ELEMENT_ID columns are required.
  -- TEXT and TYPE are not required but highly recommended.
  -- EMBEDDINGS is required if embeddings are being generated.
  -- All other "metadata" columns are optional.
  CREATE TABLE <database_name>.<schema_name>.ELEMENTS (
      ID VARCHAR(36) PRIMARY KEY NOT NULL DEFAULT UUID_STRING(),
      RECORD_ID VARCHAR,
      ELEMENT_ID VARCHAR,
      TEXT VARCHAR,
      TYPE VARCHAR,
      EMBEDDINGS VECTOR(FLOAT, <number-of-dimensions>),
      IS_EXTRACTED VARCHAR,
      POINTS VARCHAR,
      SYSTEM VARCHAR,
      LAYOUT_WIDTH INTEGER,
      LAYOUT_HEIGHT INTEGER,
      FILETYPE VARCHAR,
      LANGUAGES ARRAY,
      PAGE_NUMBER VARCHAR,
      IMAGE_MIME_TYPE VARCHAR,
      FILENAME VARCHAR,
      URL VARCHAR,
      RECORD_LOCATOR VARCHAR
  );
  ```

  Unstructured cannot provide a schema that is guaranteed to work in all
  circumstances. This is because these schemas will vary based on your source files' types; how you
  want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Snowflake**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.

* **Account ID** (*required*): The target Snowflake account's identifier.

* **Role** (*required*): The name of the Snowflake role that the user belongs to. This role must have the appropriate access to the target Snowflake warehouse, database, schema, and table.

* **User** (*required*): The target Snowflake user's login name (not their username).

* **Password** (*required*): The user's programmatic access token (PAT).

  <Note>
    Specifying a password is no longer recommended, as passwords are being deprecated by Snowflake. Use a PAT instead.
  </Note>

* **Host** (*required*): The hostname of the target Snowflake warehouse.

* **Port** (*required*): The warehouse's port number. The default is `443` if not otherwise specified.

* **Database** (*required*): The name of the target Snowflake database.

* **Schema** (*required*): The name of the target Snowflake schema within the database.

* **Table** (*required* for source connector only): The name of the target Snowflake table within the database's schema. For the destination connector, the default is `elements` if not otherwise specified.

* **Columns** (source connector only): A comma-separated list of columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* **ID Column** (*required*, source connector only): The name of the column that uniquely identifies each record in the table.

* **Record ID Key** (destination connector only): The name of the column that uniquely identifies each record in the table. The default is `record_id` if not otherwise specified.

* **Batch Size** (*required*): The maximum number of rows to fetch for each batch. The default is `50` if not otherwise specified.

## Learn more

* <Icon icon="blog" />  [Powering Enterprise RAG: Unstructured's New Snowflake Integration](https://unstructured.io/blog/powering-enterprise-rag-unstructured-s-new-snowflake-integration)


# Teradata
Source: https://docs.unstructured.io/pipelines/sources/teradata-sql



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Teradata.

The requirements are as follows.

* A Teradata Vantage system that can be accessed by its host name or IP address.

  For example, a Teradata Vantage system in Teradata ClearScape Analytics Experience includes:

  * A Teradata ClearScape Analytics Experience account.
  * An environment in the account.
  * A Teradata Vantage database in the environment.
  * The name and password for a Teradata user who has the appropriate access to the database.

  [Learn how to create these in Teradata ClearScape Analytics Experience](https://developers.teradata.com/quickstarts/get-access-to-vantage/clearscape-analytics-experience/getting-started-with-csae/).

* The system's corresponding host name or IP address.

  For example, you can get these values from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics Experience account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Host** value.<br />

* The name of the target database in the system. To get a list of available databases in the system, you can run a Teradata SQL query such as the following:

  ```sql theme={null}
  SELECT DatabaseName 
  FROM DBC.DatabasesV 
  ORDER BY DatabaseName;
  ```

* The name of the target table in the database. To get a list of available tables in a database, you can run a Teradata SQL query such as the following, replacing `<database-name>` with the name of the target database:

  ```sql theme={null}
  SELECT TableName
  FROM DBC.TablesV
  WHERE DatabaseName = '<database-name>' AND TableKind = 'T'
  ORDER BY TableName;
  ```

  When Unstructured writes rows to a table, the table's columns must have a schema that is compatible with Unstructured.
  Unstructured cannot provide a schema that is guaranteed to work for everyone in all circumstances.
  This is because these schemas will vary based on
  your source files' types; how you want Unstructured to partition, chunk, and generate embeddings;
  any custom post-processing code that you run; and other factors.

  In any case, note the following about table schemas:

  * The following columns are always required by Unstructured: `record_id` and `element_id`.
  * The following columns are optional for Unstructured, but highly recommended: `text` and `type`.
  * The rest of the columns are optional and typically will be output by Unstructured as part of the `metadata` field.
  * If Unstructured is generating vector embeddings, the `embeddings` column is also required.

    <Warning>
      The destination connector outputs Unstructured-generated [embeddings](/concepts/embedding) that are not directly compatible
      with Teradata Enterprise Vector Store. To use embeddings with Teradata Enterprise Vector Store, Unstructured
      recommends that you choose from among the following options:

      * Define a column in your target table named `embeddings` that is of type `VARCHAR(64000)`, to store the
        Unstructured-generated embeddings. After Unstructured adds its embeddings to your `embeddings` column,
        choose from among Teradata's options to convert the `embeddings` column's `VARCHAR` values to the
        Teradata [VECTOR Data Type](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/VECTOR-Data-Type)
        yourself.
      * Omit any columns named `embedding`, `message`, or `num_tokens` from your target table. Then choose from
        among Teradata's options (such as [AI\_TextEmbeddings](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-Vector-Store-User-Guide/Vector-Store-Components-and-Features/In-Database-Analytic-Functions-for-Vector-Store))
        to have Teradata generate the embeddings for you, instead of having Unstructured generate them.
    </Warning>

  Here is an example table schema that is compatible with Unstructured. It includes all of the required and recommended columns, as
  well as a few additional columns that are typically output by Unstructured as part of the `metadata` field. Be sure to replace
  `<database-name>` with the name of the target database and `<table-name>` with the name of the target table (by Unstructured convention,
  the table name is typically `elements`, but this is not a requirement).

  ```sql theme={null}
  CREATE SET TABLE "<database-name>"."<table-name>" (
    "id" VARCHAR(64) NOT NULL,
    PRIMARY KEY ("id"),
    "record_id" VARCHAR(64),
    "element_id" VARCHAR(64),
    "text" VARCHAR(32000) CHARACTER SET UNICODE,
    "type" VARCHAR(50),
    "embeddings" VARCHAR(64000), -- Add this column only if Unstructured is generating vector embeddings.
    "last_modified" VARCHAR(50),
    "languages" VARCHAR(200),
    "file_directory" VARCHAR(500),
    "filename" VARCHAR(255),
    "filetype" VARCHAR(50),
    "record_locator" VARCHAR(1000),
    "date_created" VARCHAR(50),
    "date_modified" VARCHAR(50),
    "date_processed" VARCHAR(50),
    "permissions_data" VARCHAR(1000),
    "filesize_bytes" INTEGER,
    "parent_id" VARCHAR(64)
  )
  ```

* For the source connector, the name of the primary key column in the table (for example, a column named `id`, typically defined as `"id" VARCHAR(64) NOT NULL, PRIMARY KEY ("id")`).

* For the source connector, the names of any specific columns to fetch from the table. By default, all columns are fetched unless otherwise specified.

* For the destination connector, the name of the column in the table that uniquely identifies each record for Unstructured to perform any necessary record updates. By default convention, Unstructured expects this field to be named `record_id`.

* The name of the Teradata user who has the appropriate access to the target database.

  For example, you can get this from Teradata ClearScape Analytics Experience as follows:

  1. Sign in to your Teradata ClearScape Analytics account.<br />
  2. On the sidebar, under **Environments**, click the name of the database's corresponding environment.<br />
  3. Under **Connection details for Vantage database**, use the **Username** value.<br />

  The Teradata SQL command to get a list of available users is as follows:

  ```sql theme={null}
  SELECT UserName 
  FROM DBC.UsersV 
  ORDER BY UserName;
  ```

* The password for the user, which was set up when the user was created.

  If the user has forgotten their password, the Teradata SQL command to change a user's password is as follows, replacing `<user-name>` with the name of the user and `<new-password>` with the new password:

  ```sql theme={null}
  MODIFY USER <user-name> SET PASSWORD = '<new-password>';
  ```

  To change a user's password, you must be an administrator (such as the `DBC` user or another user with `DROP USER` privileges).

<Info>
  The `record_id`, `element_id`, and `id` fields are closely related, but each has a distinct purpose. For more information, see [How connectors use record IDs, element IDs, and IDs](/api-reference/record-element-id).
</Info>

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Teradata**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Host** (*required*): The hostname or IP address associated with the target Teradata Vantage database.
* **Database**: The name of the target database. By default, the default database name is used if not otherwise specified. To get the name of the default database, you can run the Teradata SQL command `SELECT DATABASE;`.
* **Table Name** (*required*): The name of the target table in the database.
* **Batch Size**: The maximum number of rows per batch. The default is `100` if not otherwise specified.
* **ID Column** (*required*, source connector only): The name of the primary key column that Teradata uses to uniquely identify each record in the table.
* **Record ID Key** (destination connector only): The name of the column that Unstructured uses to uniquely identify each record in the table for record update purposes. The default is `record_id` if not otherwise specified.
* **Columns** (source connector only): The names of the columns to fetch from the table, seperated by commas. By default, all columns are fetched unless otherwise specified.
* **Username** (*required*): The name of the user who has the appropriate access to the database.
* **Password** (*required*): The password for the user.


# VAST
Source: https://docs.unstructured.io/pipelines/sources/vast

Ingest your files into Unstructured from [VAST](https://www.vastdata.com/).

<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

## Requirements

You will need:

* A [VAST](https://www.vastdata.com/) user account with permissions to the bucket to which you want to connect.

* A bucket that contains the objects to which you want to connect. A bucket is a storage resource that uses Amazon's Simple Storage Service (S3) as an access protocol.

  To create a bucket, see [Creating S3 Buckets](https://kb.vastdata.com/documentation/docs/creating-s3-buckets-via-vms-1) in the *VAST Cluster Administrator's Guide*.

* The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/` (for example, `vast://my-bucket/`). If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/` (for example, `vast://my-bucket/my-folder/`).

* The URL for the VAST region in which the bucket is located (for example, `https://vast-cluster.my-company.com`).

* An enabled access and secret key pair that grants the authenticated user permission to the VAST bucket.

  To grant a user access to a bucket, and attach identity policies that define their permissions on that bucket, see [Managing S3 User Access from the VAST Web UI](https://kb.vastdata.com/documentation/docs/managing-s3-user-access-from-the-vast-web-ui-1) or [Managing S3 Access from the VAST CLI](https://kb.vastdata.com/documentation/docs/managing-s3-access-from-the-vast-cli) in the *VAST Cluster Administrator's Guide*.

  <Note>
    VAST recommends using identity policies and bucket policies to control S3 bucket access.

    * *Identity policies* are attached to users and groups, and defines the actions the users or groups can perform against buckets.
    * *Bucket policies* are attached to a view, and define actions that particular users or groups can perform against the view and files or directories under the view.

    For more information, see [Overview of S3 Access Management](https://kb.vastdata.com/documentation/docs/overview-of-s3-access-management) in the *VAST Cluster Administrator's Guide*.

    Identity and bucket policies support a subset of elements listed in [Amazon's IAM JSON Policy Reference](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies.html). For information about the required JSON format, supported elements and identity and bucket policy examples, see [Identity and Bucket Policy Reference](https://kb.vastdata.com/documentation/docs/identity-and-bucket-policy-reference-1) in the *VAST Cluster Administrator's Guide*.
  </Note>

  For source connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:ListBucket`
  * `s3:GetObject`

  For destination connectors, the authenticated user must have bucket permissions for the following S3 actions:

  * `s3:PutObject`

VAST buckets are private by default. [Anonymous access](https://kb.vastdata.com/documentation/docs/overview-of-vast-cluster-s3-implementation-1#support-for-authenticated-or-anonymous-requests) can be explicitly enabled using bucket policies, but is not recommended.

## Create the connector

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **VAST**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for this connector.
* **Remote URL** (*required*): The path to the files in the bucket.

  If the target files are in the root of the bucket, the path should be formatted as `vast://bucket/`. For example: `vast://my-bucket/`

  If the target files are in a folder, format the path to the target folder as `vast://bucket/path/to/folder/`. For example: `vast://my-bucket/my-folder/`

  <Note>
    For destination connectors, the relative directory structure of files is preserved when files are uploaded. Also, files are overwritten if they already exist at the destination.
  </Note>
* **Recursive** (source connector only): Check this box to access sub-folders within the bucket.
* **Max Objects** (source connector only): The maximum number of objects to index. This is useful for large buckets.
* **Endpoint URL** (*required*): The VAST storage endpoint URL. For example, `https://vast-cluster.my-company.com`.
* **Key** (*required*): The access key ID for the authenticated user.
* **Secret** (*required*): The secret access key corresponding to the preceding access key ID.


# Zendesk
Source: https://docs.unstructured.io/pipelines/sources/zendesk



<Note>
  First time creating a connector? [Read this first](/pipelines/connector-first-time-reqs).
</Note>

Ingest your files into Unstructured from Zendesk.

The requirements are as follows.

<iframe title="YouTube video player" />

* A [Zendesk account](https://www.zendesk.com/register/).
* Your [Zendesk subdomain](https://support.zendesk.com/hc/en-us/articles/4409381383578-Where-can-I-find-my-Zendesk-subdomain), for example, the `<organization>` part of `<organization>.zendesk.com`.
* The login email address for your Zendesk account.
* An [API token](https://support.zendesk.com/hc/en-us/articles/4408889192858-Managing-access-to-the-Zendesk-API#topic_tcb_fk1_2yb) (not an OAuth token) for your login email address.

To create the source connector:

1. On the sidebar, click **Connectors**.
2. Click **Sources**.
3. Cick **New** or **Create Connector**.
4. Give the connector some unique **Name**.
5. In the **Provider** area, click **Zendesk**.
6. Click **Continue**.
7. Follow the on-screen instructions to fill in the fields as described later on this page.
8. Click **Save and Test**.

Fill in the following fields:

* **Name** (*required*): A unique name for the connector.
* **Subdomain** (*required*): The subdomain of your Zendesk site, for example the `<organization>` part of `<organization>.zendesk.com`.
* **Email** (*required*): The email address of the user account that has access to the Zendesk subdomain.
* **API Token** (*required*): The API token (not an OAuth token) for the user account. For more information, see [Managing access to the Zendesk API](https://support.zendesk.com/hc/en-us/articles/226022787-Generating-a-new-API-token).
* **Item type**: The type of item to parse. Allowed values include `tickets` and `articles`. If no value is provided, the default is `tickets`.
* **Batch size**: The maximum number of items to structure per batch. The default is `2`.

## Learn more

* <Icon icon="blog" />  [Unlock Customer Support Knowledge: Unstructured Integrates with Zendesk](https://unstructured.io/blog/unlock-customer-support-knowledge-unstructured-integrates-with-zendesk)


# Supported file types
Source: https://docs.unstructured.io/pipelines/supported-file-types



Unstructured Pipelines supports processing of the following file types:

By file extension:

| File extension |
| -------------- |
| `.abw`         |
| `.bmp`         |
| `.csv`         |
| `.cwk`         |
| `.dbf`         |
| `.dif`\*       |
| `.doc`         |
| `.docx`        |
| `.dot`         |
| `.dotm`        |
| `.eml`         |
| `.epub`        |
| `.et`          |
| `.eth`         |
| `.fods`        |
| `.heic`        |
| `.htm`         |
| `.html`        |
| `.hwp`         |
| `.jpeg`        |
| `.jpg`         |
| `.md`          |
| `.mcw`         |
| `.msg`         |
| `.mw`          |
| `.odt`         |
| `.org`         |
| `.p7s`         |
| `.pbd`         |
| `.pdf`         |
| `.png`         |
| `.pot`         |
| `.ppt`         |
| `.pptm`        |
| `.pptx`        |
| `.prn`         |
| `.rst`         |
| `.rtf`         |
| `.sdp`         |
| `.sxg`         |
| `.tif`         |
| `.tiff`        |
| `.txt`         |
| `.tsv`         |
| `.xls`         |
| `.xlsx`        |
| `.xml`         |
| `.zabw`        |

By file type:

| Category          | File types                                                        |
| ----------------- | ----------------------------------------------------------------- |
| Apple             | `.cwk`, `.mcw`                                                    |
| CSV               | `.csv`                                                            |
| Data Interchange  | `.dif`\*                                                          |
| dBase             | `.dbf`                                                            |
| E-mail            | `.eml`, `.msg`, `.p7s`                                            |
| EPUB              | `.epub`                                                           |
| HTML              | `.htm`, `.html`                                                   |
| Image             | `.bmp`, `.heic`, `.jpeg`, `.jpg`, `.png`, `.prn`, `.tif`, `.tiff` |
| Markdown          | `.md`                                                             |
| OpenOffice        | `.odt`                                                            |
| Org Mode          | `.org`                                                            |
| Other             | `.eth`, `.pbd`, `.sdp`                                            |
| PDF               | `.pdf`                                                            |
| Plain text        | `.txt`                                                            |
| PowerPoint        | `.pot`, `.ppt`, `.pptm`, `.pptx`                                  |
| reStructured Text | `.rst`                                                            |
| Rich Text         | `.rtf`                                                            |
| Spreadsheet       | `.et`, `.fods`, `.mw`, `.xls`, `.xlsx`                            |
| StarOffice        | `.sxg`                                                            |
| TSV               | `.tsv`                                                            |
| Word processing   | `.abw`, `.doc`, `.docx`, `.dot`, `.dotm`, `.hwp`, `.zabw`         |
| XML               | `.xml`                                                            |

`*` For `.dif`, `\n` characters in `.dif` files are supported, but `\r\n` characters will raise the error
`UnsupportedFileFormatError: Partitioning is not supported for the FileType.UNK file type`.


# Unstructured Pipelines walkthrough
Source: https://docs.unstructured.io/pipelines/walkthrough



This walkthrough provides you with deep, hands-on experience with the [Unstructured Pipelines](/pipelines/overview). As you follow along, you will learn how to use many of Unstructured's
features for [partitioning](/concepts/partitioning), [enriching](/concepts/enriching/overview), [chunking](/concepts/chunking), [embedding](/concepts/embedding), and [structured data extraction](/concepts/structured-data-extractor/data-extractor). These features are optimized for turning
your source documents and data into information that is well-tuned for
[retrieval-augmented generation (RAG)](https://unstructured.io/blog/rag-whitepaper),
[agentic AI](https://unstructured.io/problems-we-solve#powering-agentic-ai),
and [model fine-tuning](https://www.geeksforgeeks.org/deep-learning/what-is-fine-tuning/).

This walkthrough uses two sample files to demonstrate how Unstructured identifies and processes content such as image, graphs, complex tables, non-English characters, and handwriting.
These files, which are available for you to download to your local machine, include:

* Wang, Z., Liu, X., & Zhang, M. (2022, November 23).
  *Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modeling*.
  arXiv.org. [https://arxiv.org/pdf/2211.12781](https://arxiv.org/pdf/2211.12781). This 12-page PDF file features English and non-English characters, images, graphs, and complex tables.
  Throughout this walkthrough, this file's title is shortened to "Chinese Characters" for brevity.
* United States Central Security Service. (2012, January 27). *National Cryptologic Museum Opens New Exhibit on Dr. John Nash*.
  United States National Security Agency. [https://courses.csail.mit.edu/6.857/2012/files/H03-Cryptosystem-proposed-by-Nash.pdf](https://courses.csail.mit.edu/6.857/2012/files/H03-Cryptosystem-proposed-by-Nash.pdf).
  This PDF file features English handwriting and scanned images of documents.
  Throughout this walkthrough, this file's title is shortened to "Nash letters" for brevity.

<Note>
  This walkthrough focuses on local files for ease-of-use demonstration purposes.

  This walkthrough does not cover how to use
  Unstructured to set up [connectors](/pipelines/connectors) to do large-scale batch processing of multiple files and semi-structured data that are stored in remote locations.
  To learn how to set up connectors and do large-scale batch processing later, see the [next steps](#next-steps) after you finish this walkthrough.
</Note>

If you are not able to complete any of the following steps, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

<Tip>
  *What are these green boxes?*

  As you move through this walkthrough, you will notice tips like this one. These tips are designed to help expand
  your knowledge about Unstructured as you go. Feel free to skip these tips for now if you are in a hurry. You can always return to them later to learn more.
</Tip>

## Step 1: Sign up and sign in to Unstructured

Let's get started!

1. If you do not already have an Unstructured account, [sign up for free](https://unstructured.io/?modal=try-for-free).
   After you sign up, you are automatically signed in to your new Unstructured **Let's Go** account, at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     To sign up for a **Business** account instead, [contact Unstructured Sales](https://unstructured.io/?modal=contact-sales), or [learn more](/pipelines/overview#how-am-i-billed%3F).
   </Note>

2. If you have an Unstructured **Let's Go**, **Pay-As-You-Go**, or **Business SaaS** account and are not already signed in, sign in to your account at [https://platform.unstructured.io](https://platform.unstructured.io).

   <Note>
     For other types of **Business** accounts, see your Unstructured account administrator for sign-in instructions,
     or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

## Step 2: Create a custom workflow

In this step, you create a custom [workflow](/pipelines/workflows) in your Unstructured account.

Workflows are defined sequences of processes that automate the flow of data from your source documents and data into Unstructured for processing.
Unstructured then sends its processed data over into your destination file storage locations, databases, and vector stores. Your
RAG apps, agents, and models can then use this processed data in those destinations to do things more quickly and accurately such as
[answering users' questions](https://learn.microsoft.com/en-us/azure/developer/ai/advanced-retrieval-augmented-generation),
[automating business processes](https://unstructured.io/problems-we-solve#business-process-automation),
and [expanding your organization's available body of knowledge](http://knowledgemanagement.ie/the-critical-role-of-knowledge-management-as-a-foundation-for-llms-and-ai/).

<Tip>
  *Which kinds of sources and destinations does Unstructured support?*

  Unstructured can connect to many types of sources and destinations including file storage services such as Amazon S3 and Google Cloud Storage;
  databases such as PostgreSQL; and vector storage and database services such as MongoDB Atlas and Pinecone.

  See the full list of [supported source and destination connectors](/pipelines/connectors).
</Tip>

<Tip>
  *Which kinds of files does Unstructured support?*

  Unstructured can process a wide variety of file types including PDFs, word processing documents, spreadsheets, slide decks, HTML, image files, emails, and more.

  See the full list of [supported file types](/pipelines/supported-file-types).
</Tip>

Let's get going!

1. After you are signed in to your Unstructured account, on the sidebar, click **Workflows**.

   <img alt="Workflows button on the sidebar" />

   <Tip>
     *What do the other buttons on the sidebar do?*

     * **Start** takes you to the Pipelines home page. The home page features a simple way to process one local file at a time with limited default settings. [Learn how](/welcome#unstructured-pipelines-quickstart).
     * **Connectors** allows you to create and manage your [source](/pipelines/sources/overview) and
       [destination](/pipelines/destinations/overview) connectors.
     * **Jobs** allows you to see the results of your workflows that are run manually (once) and automatically (on a regular time schedule). [Learn more](/pipelines/jobs).
     * **API Keys** allows you to use code to create and manage connectors, workflows, and jobs programmatically instead of by using the UI. [Learn more](/pipelines/account/api-key-url).
     * Your user icon at the bottom of the sidebar allows you to manage your Unstructured account. You can also sign out of your account from here. [Learn more](/pipelines/account/overview).
   </Tip>

2. Click **New Workflow**.

   <img alt="New Workflow button" />

3. With **Build it Myself** already selected, click **Continue**.

   <img alt="Build it Myself workflow option" />

   <Tip>
     *What does **Build it For Me** do?*

     The **Build it For Me** option creates an [automatic workflow](/pipelines/workflows#create-an-automatic-workflow) with limited, sensible default settings to enable you to get good-quality results faster. However,
     this option requires that you first have an existing remote source and destination connector to add to the workflow. To speed things up here and keep things simple,
     this walkthrough only processes files from your local machine and skips the use of connectors. To learn how to use connectors later,
     see the [next steps](#next-steps) at the end of this walkthrough.
   </Tip>

4. The workflow designer appears.

   <img alt="The workflow designer" />

   <Tip>
     *What are all the parts of the workflow designer?*

     The middle portion of the workflow designer is the workflow *directed acyclic graph* (DAG), which contains a
     collection of *nodes* connected by *edges* that go in only one direction. You can think of the DAG similar to a flowchart
     for a process. *Directed* means the arrows show the flow from one step to the next, and *acyclic* means you cannot
     follow the arrows backward to get back to the starting point.

     The workflow settings pane on the right includes the following tabs:

     * The **Details** tab allows you to see when this workflow was created and which jobs were run for this workflow.
     * **Schedule** allows you to set up a schedule for this workflow to run automatically (on a regular time schedule).
     * **Settings** allows you to specify whether every time this workflow runs, that Unstructured's results will overwrite any previous results in the destination location.
       To turn on this behavior, check the **Overwrite existing results** box. To turn it off, uncheck the box. Note that this setting works only
       for blob storage destination connectors such as the ones for Amazon S3, Azure Blob Storage, and Google Cloud Storage.
     * **FAQ** contains additional information about how to use the workflow designer.

     If the workflow settings pane is not visible, click the **Settings** button near the bottom to show it.

     There are also buttons near the bottom to undo or redo recent edits to the workflow DAG, zoom in and out of the workflow
     designer, re-center the DAG within the designer, and add a new node to the DAG.

     To change this workflow's name, click the workflow's name in the title bar, type a new name, and then press **Enter**.
   </Tip>

## Step 3: Experiment with partitioning

In this step, you use your new workflow to [partition](/concepts/partitioning) the sample PDF files that you downloaded earlier onto your local machine.
Partitioning is the process where Unstructured identifies and extracts content from your source documents and then
outputs this content as a series of contextually-rich [document elements and metadata](/concepts/document-elements), which are
well-tuned for RAG, agentic AI, and model fine-tuning. This step
shows how well Unstructured's **VLM** partitioning strategy handles challenging content such as complex tables, multilanguage characters, and handwriting.

1. With the workflow designer active from the previous step, at the bottom of the **Source** node, click **Drop file to test**.

   <img alt="Drop file to test button" />

2. Browse to and select the "Chinese Characters" PDF file that you downloaded earlier.

3. In the workflow designer, click the **Partitioner** node and then, in the node's settings pane's **Details** tab, select **VLM**.<br />

   <Tip>
     *When would I choose **Auto**, **Fast**, **High Res**, or **VLM**?*

     * **Auto** is recommended in most cases. It lets Unstructured figure out the best strategy to switch over to for each incoming file (and even for each page if the incoming file is a PDF), so you don't have to!
     * **Fast** is only for when you know for certain that none of your files have tables, images, or multilanguage, scanned, or handwritten content in them. It's optimized for partitioning text-only content and is the fastest of all the strategies. It can recognize the text for only a few languages other than English.
     * **High Res** is only for when you know for certain that at least one of your files has images or simple tables in them, and that none of your files also have scanned or handwritten content in them. It can recognize the text for more languages than **Fast** but not as many as **VLM**.
     * **VLM** is great for any file, but it is best when you know for certain that some of your files have a combination of tables (especially complex ones), images, and multilanguage, scanned, or handwritten content. It's the highest quality but slowest of all the strategies.
   </Tip>

4. Under **Select VLM Model**, under **Vertex AI**, select **Gemini 2.0 Flash**.<br />

   <img alt="Selecting the VLM for partitioning" />

   <Note>
     The list of available models shown is constantly being updated. Your list might also differ from the preceding screenshot, depending on your Unstructured
     account type. If **Gemini 2.0 Flash** is not available, choose another available model from the list. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

     If you have an Unstructured **Business** account and want to add more models to this list, contact your
     Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
     [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

   <Tip>
     *When I choose **VLM**, when would I choose one of these models over another?*

     A *vision language model* (VLM) is designed to use sophisticated AI techniques and logic to combine advanced image and text understanding, resulting in more accurate and
     contextually-rich output.

     As VLMs are constantly being released and improved, Unstructured is always adding to and updating its list of supported VLMs.
     If you aren't getting consistent results with one VLM for a particular set of files, switching over to another one might improve your results, depending on that VLM's capabilities and the sample data that it was trained on.
   </Tip>

5. Click **Test**.<br />

   <img alt="Begin testing the local file" />

6. The PDF file appears in a pane on the left side of the screen, and Unstructured's output appears in a **Test output** pane on the right side of the screen.

   <img alt="Showing the test output results" />

   <Tip>
     *What am I looking at in the output here?*

     * Unstructured outputs its results in industry-standard [JSON](https://www.json.org) format, which is ideal for RAG, agentic AI, and model fine-tuning.
     * Each object in the JSON is called a [document element](/concepts/document-elements) and contains a `text` representation of the content that Unstructured detected for the particular portion of the document that was analyzed.
     * The `type` is the kind of document element that Unstructured categorizes it as, such as whether it is a title (`Title`), a table (`Table`), an image (`Image`), a series of well-formulated sentences
       (`NarrativeText`), some kind of free text (`UncategorizedText`), a part of a list (`ListItem`), and so on. [Learn more](/concepts/document-elements#element-type).
     * The `element_id` is a unique identifier that Unstructured generates to refer to each document element. [Learn more](/concepts/document-elements#element-id).
     * `metadata` contains supporting details about each document element, such as the page number it occurred on, the file it occurred in, and so on. [Learn more](/concepts/document-elements#metadata).
   </Tip>

   <Tip>
     *What else can I do here?*

     * You can scroll through the original file on the left or, where supported for a given file type, click the up and down arrows to page through the file one page at a time.
     * You can scroll through Unstructured's JSON output on the right, and you can click **Search JSON** to search for specific text in the JSON output. You will do this next.
     * **Download Full JSON** allows you to download the full output to your local machine as a JSON file.
     * **View JSON at this step** allows you to view the JSON output at each step in the workflow as it was further processed. There's only one step right now (the **Partitioner** step),
       but as you add more nodes to the workflow DAG, this can be a useful tool to see how the JSON output changes along the way.
     * The close (**X**) button returns you to the workflow designer.
   </Tip>

7. Notice the following in the JSON output, which you can get to by clicking **Search JSON** above the output:

   <img alt="Searching the JSON output" />

   * The Chinese characters on page 1. Search for the text `verbs. The characters`. Notice how the Chinese characters are output. We'll see accuracy improvements to this output later in Step 4 in the enrichments portion of this walkthrough.
   * The tables on pages 1, 6, 7, 8, 9, and 12. Search for the text `"Table"` (including the quotation marks) to see how the VLM interprets the various tables. We'll see changes to these elements' `text` and `metadata.text_as_html` contents later in Step 4 in the enrichments portion of this walkthrough.
   * The images on pages 3, 7, and 8. Search for the text `"Image"` (including the quotation marks) to see how the VLM interprets the various images. We'll see changes to these elements' `text` contents later in Step 4 in the enrichments portion of this walkthrough.

   <Note>
     If **Search JSON** is not clickable, this is probably because the JSON output is too large for the online viewer.
     Click **Download full JSON** and open the downloaded JSON file in an offline text editor (such as Visual Studio Code).
     [Learn more](/examplecode/tools/json-tools#visual-studio-code).
   </Note>

8. Now try looking at the "Nash letters" PDF file's output. To do this:

   a. Click the close (**X**) button above the output on the right side of the screen.<br />
   b. At the bottom of the **Source** node, click the existing PDF's file name.<br />
   c. Browse to and select the "Nash letters" file that you downloaded earlier to your local machine.<br />
   d. Click **Test**.<br />

9. Notice the following in the JSON output:

   * The handwriting on page 3. Search for the text `I have written RAND`. Notice how well the handwriting is recognized.
   * The mimeograph on page 18. Search for the text `The system which`. Notice how well the mimeographed content is recognized.

10. When you are done, be sure to click the close (**X**) button above the output on the right side of the screen, to return to
    the workflow designer for the next step.

## Step 4: Experiment with enriching

In this step, you add several [enrichments](/concepts/enriching/overview) to your workflow, such as generating summary descriptions of detected images and tables,
HTML representations of detected tables, and detected entities (such as people and organizations) and the inferred relationships among these entities.

1. With the workflow designer active from the previous step, change the **Partitioner** node to use **High Res**.

2. Between the **Partitioner** and **Destination** nodes, click the add (**+**) icon, and then click **Enrich > Image Description**.

   <img alt="Adding an image description enrichment node" />

3. In the node's settings pane's **Details** tab, click:

   * Any available choice for **Provider** (for example, **Anthropic**).
   * Any available choice for **Model** (for example, **Claude Sonnet 4.5** if you chose **Anthropic** for **Provider**).

   <Note>
     The lists of available models for these enrichments are constantly being updated. Your lists might also differ from these instructions, depending on your Unstructured
     account type. If the suggested model for an enrichment is not available, choose another available model from the list. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

     If you have an Unstructured **Business** account and want to add more models to these lists, contact your
     Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
     [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

   <Tip>
     The image description enrichment generates a summary description of each detected image. This can help you to more quickly and easily understand
     what each image is all about without having to stop to manually visualize and interpret the image's content yourself. This also provides additional helpful context about the image for your RAG apps, agents, and models. [Learn more](/concepts/enriching/image-descriptions).
   </Tip>

4. Repeat this process to add four more nodes between the **Partitioner** and **Destination** nodes, as follows:

   a. Add a table description enrichment (click the add (**+**) icon to the right of the preceding node, and then click **Enrich > Table Description**).

   In the node's settings pane's **Details** tab, click:

   * Any available choice for **Provider** (for example, **Anthropic**).
   * Any available choice for **Model** (for example, **Claude Sonnet 4.5** if you chose **Anthropic** for **Provider**).

   <Tip>
     The table description enrichment generates a summary description of each detected table. This can help you to more quickly and easily understand
     what each table is all about without having to stop to manually read through the table's content yourself. This also provides additional helpful context about the table for your RAG apps, agents, and models. [Learn more](/concepts/enriching/table-descriptions).
   </Tip>

   b. Add a table to HTML enrichment (click the add (**+**) icon to the right of the preceding node, and then click **Enrich > Table to HTML**).

   In the node's settings pane's **Details** tab, click:

   * If you choose **Standard** for **Mode**, any available choice for **Provider** (for example, **Anthropic**).
   * If you choose **Standard** for **Mode**, any available choice for **Model** (for example, **Claude Sonnet 4.5** if you chose **Anthropic** for **Provider**).
   * If you choose **Agentic** for **Mode**, a default provider and model are automatically selected for you.

   <Tip>
     The table to HTML enrichment generates an HTML representation of each detected table. This can help you to more quickly and accurately recreate the table's content elsewhere later as needed. This also provides additional context about the table's structure for your RAG apps, agents, and models.
     The **Agentic** mode uses agentic AI to generate the table-to-HTML output, which typically provides more accurate output for complex tables than the **Standard** mode. [Learn more](/concepts/enriching/table-to-html).
   </Tip>

   c. Add a named entity recognition enrichment (click the add (**+**) icon to the right of the preceding node, and then click **Enrich > NER**).

   In the node's settings pane's **Details** tab, click:

   * Any available choice for **Provider** (for example, **Anthropic**).
   * Any available choice for **Model** (for example, **Claude Sonnet 4.5** if you chose **Anthropic** for **Provider**).

   <Tip>
     The named entity recognition (NER) enrichment generates a list of detected entities (such as people and organizations) and the inferred relationships among these entities. This provides additional context about these entities' types and their relationships for your graph databases, RAG apps, agents, and models. [Learn more](/concepts/enriching/ner).
   </Tip>

   d. Add a generative OCR enrichment (click the add (**+**) icon to the right of the preceding node, and then click **Enrich > Generative OCR**).

   In the node's settings pane's **Details** tab, click:

   * Any available choice for **Provider** (for example, **Anthropic**).
   * Any available choice for **Model** (for example, **Claude Sonnet 4.5** if you chose **Anthropic** for **Provider**).

   <Tip>
     The generative OCR enrichment improves, as needed, the accuracy of text blocks that Unstructured initially processed during its partitioning phase. [Learn more](/concepts/enriching/generative-ocr).
   </Tip>

   <Warning>
     Generative OCR does not process any text blocks by default. You must also explicitly specify which document element
     types containing text that you want generative OCR to process. To do this:

     1. Click the **Partitioner** node.
     2. In the node's settings pane, scroll down to and then click a blank area inside of the **Extract Image Block Types** list.
     3. Select each [document element type](/concepts/document-elements#element-type) that you want generative OCR to process. For this
        walkthrough, select only **NarrativeText**.

     Generative OCR does not process the text of any `Image` or `Table` elements if they have already been processed by
     [image description](/api-reference/workflow/nodes/enrichment/enrichment-image-description) or [table description](/api-reference/workflow/nodes/enrichment/enrichment-table-description) enrichments, respectively. Do
     not remove the **Image** or **Table** document elements types from this **Extract Image Block Types** list, or else
     the image description and table description enrichments in your workflow might produce unexpected results or might not work at all.
   </Warning>

   The workflow designer should now look similar to this:

   <img alt="The workflow with enrichments added" />

5. Change the **Source** node to use the "Chinese Characters" PDF file, and then click **Test**.

6. In the **Test output** pane, make sure that **Generative OCR (6 of 6)** is showing. If not, click the right arrow (**>**) until **Generative OCR (6 of 6)** appears, which will show the output from the last node in the workflow.

7. Some interesting portions of the output include the following:

   * The Chinese characters on page 1. Search again for the text `verbs. The characters`. Notice how the accuracy of the Chinese character output is improved.
   * The images on pages 3, 7, and 8. Search again for the text `"Image"` (including the quotation marks). Notice the summary description for each image.
   * The tables on pages 1, 6, 7, 8, 9, and 12. Search again for the text `"Table"` (including the quotation marks). Notice the summary description for each of these tables.
     Also notice the `text_as_html` field for each of these tables.
   * The identified entities and inferred relationships among them. Search for the text `Zhijun Wang`. Of the eight instances of this name, notice
     the author's identification as a `PERSON` three times, the author's `published` relationship twice, and the author's `affiliated_with` relationship twice.

8. When you are done, be sure to click the close (**X**) button above the output on the right side of the screen, to return to
   the workflow designer for the next step.

## Step 5: Experiment with chunking

In this step, you apply [chunking](/concepts/chunking) to your workflow. Chunking is the process where Unstructured rearranges
the resulting document elements' `text` content into manageable "chunks" to stay within the limits of an AI model and to improve retrieval precision.

<Tip>
  *What kind of chunking strategy should I use, and how big should my chunks be?*

  Unfortunately, there is no one-size-fits-all answer to this question. However, there are some general considerations and guidelines that can help you to determine
  the best chunking strategy and chunk size for your specific use case. Be sure of course to also consult the documentation for your target AI model and downstream application toolsets.

  Is your content primarily organized by title, by page, by interrelated subject matter, or none of these? This can help you determine whether a
  by-title, by-page, by-similarity, or basic (by-character) chunking strategy is best. (You'll experiment with each of these strategies here later.)

  If your chunks are too small, they might lose necessary context, leading to the model providing inaccurate, irrelevant, or hallucinated results.
  On the other hand, if your chunks are too large, the model can struggle with the sheer volume of information, leading to information overload, diluted meaning, and potentially higher processing costs.
  You should aim to find a balance between chunks that are big enough to contain meaningful information, while small enough to enable performant applications and low latency responses.

  For example, smaller chunks of 128 or 256 tokens might be sufficient for capturing more granular semantic information, while larger chunks of 512 or 1024 tokens might be better for retaining more context.
  It's important here to note that *tokens* and *characters* are not the same thing! In terms of
  characters, for English text, a common approximation is 1 token being equal to about 3 or 4 characters or three-quarters of a word. Many AI model providers publish their own token-to-character calculators online that you can use for estimation purposes.

  You should experiment with a variety of chunk sizes, taking into account the kinds of content, the length and complexity of user queries and agent tasks,
  the intended end use, and of course the limits of the models you are using. Try different chunking strategies and sizes with your models and evaluate the results for yourself.
</Tip>

1. With the workflow designer active from the previous step, just before the **Destination** node, click the add (**+**) icon, and then click **Enrich > Chunker**.

   <img alt="Adding a chunker node" />

2. In the node's settings pane's **Details** tab, select **Chunk by Character**.

3. Under **Chunk by Character**, specify the following settings:

   * Check the box labelled **Include Original Elements**.
   * Set **Max Characters** to **500**.
   * Set **New After N Characters** to **400**.
   * Set **Overlap** to **50**.
   * Leave **Contextual Chunking** turned off and **Overlap All** unchecked.

   <img alt="Setting up the Chunk by Character strategy" />

   <Tip>
     *What do each of these chunking settings do?*

     * **Contextual Chunking** prepends chunk-specific explanatory context to each chunk, which has been shown to yield significant improvements in downstream retrieval accuracy. [Learn more](/concepts/chunking#contextual-chunking).
     * **Include Original Elements** outputs into each chunk's `metadata` field's `orig_elements` value the elements that were used to form that particular chunk. [Learn more](/concepts/chunking#include-original-elements-setting).
     * **Max Characters** is the "hard" or maximum number of characters that any one chunk can contain. Unstructured cannot exceed this number when forming chunks. [Learn more](/concepts/chunking#max-characters-setting).
     * **New After N Characters**: is the "soft" or approximate number of characters that any one chunk can contain. Unstructured can exceed this number if needed when forming chunks (but still cannot exceed the **Max Characters** setting). [Learn more](/concepts/chunking#new-after-n-characters-setting).
     * **Overlap**, when applied (see **Overlap All**), prepends to the current chunk the specified number of characters from the previous chunk, which can help provide additional context about this chunk relative to the previous chunk. [Learn more](/concepts/chunking#overlap-setting)
     * **Overlap All** applies the **Overlap** setting (if greater than zero) to all chunks. Otherwise, unchecking this box means that the **Overlap** setting (if greater than zero) is applied only in edge cases where "normal" chunks cannot be formed by combining whole elements. Check this box with caution as it can introduce noise into otherwise clean semantic units. [Learn more](/concepts/chunking#overlap-all-setting).
   </Tip>

4. With the "Chinese Characters" PDF file still selected in the **Source** node, click **Test**.

5. In the **Test output** pane, make sure that **Chunker (7 of 7)** is showing. If not, click the right arrow (**>**) until **Chunker (7 of 7)** appears, which will show the output from the last node in the workflow.

6. To explore the chunker's results, search for the text `"CompositeElement"` (including the quotation marks).

   <Tip>
     *In the chunked output, where did all of the document elements I saw before, such as `Title`, `Image`, and `Table`, go?*

     During chunking, the document elements that were generated during partitioning are now chunked. Because some of these document elements can be split into multiple chunks
     or combined with other chunks, these chunked document elements are now of type `CompositeElement` and `TableChunk`.

     You can have Unstructured also output the original document elements that
     these chunks were derived from by putting them into each chunk's `metadata`. To have Unstructured do this, use the **Include Original Elements** setting, as described in the preceding tip.
   </Tip>

7. Optionally, you can try running this workflow again with the **Chunk by Title** strategy, as follows:

   a. Click the close (**X**) button above the output on the right side of the screen.<br />
   b. In the workflow designer, click the **Chunker** node and then, in the node's settings pane's **Details** tab, select **Chunk by Title**.<br />
   c. Under **Chunk by Title**, specify the following settings:

   * Check the box labelled **Include Original Elements**.
   * Set **Max Characters** to **500**.
   * Set **New After N Characters** to **400**.
   * Set **Overlap** to **50**.
   * Leave **Contextual Chunking** turned off, leave **Combine Text Under N Characters** blank, and leave **Multipage Sections** and **Overlap All** unchecked.

   <Tip>
     *What do each of the chunking settings here that were not already described in the preceding tip do?*

     * **Combine Text Under N Characters** combines elements from a section into a chunk until a section reaches a length of this many characters. [Learn more](/concepts/chunking#combine-text-under-n-characters-setting).
     * **Multipage Sections** when checked, allows sections to span multiple pages. [Learn more](/concepts/chunking#multipage-sections-setting).
   </Tip>

   d. Click **Test**.<br />
   e. In the **Test output** pane, make sure that **Chunker (7 of 7)** is showing. If not, click the right arrow (**>**) until **Chunker (7 of 7)** appears, which will show the output from the last node in the workflow.<br />
   f. To explore the chunker's results, search for the text `"CompositeElement"` (including the quotation marks). Notice that the lengths of some of the chunks that immediately
   precede titles might be shortened due to the presence of the title impacting the chunk's size.

8. Optionally, you can try running this workflow again with the **Chunk by Page** strategy, as follows:

   a. Click the close (**X**) button above the output on the right side of the screen.<br />
   b. In the workflow designer, click the **Chunker** node and then, in the node's settings pane's **Details** tab, select **Chunk by Page**.<br />
   c. Under **Chunk by Page**, specify the following settings:

   * Check the box labelled **Include Original Elements**.
   * Set **Max Characters** to **500**.
   * Set **New After N Characters** to **400**.
   * Set **Overlap** to **50**.
   * Leave **Contextual Chunking** turned off, and leave **Overlap All** unchecked.

   d. Click **Test**.<br />
   e. In the **Test output** pane, make sure that **Chunker (7 of 7)** is showing. If not, click the right arrow (**>**) until **Chunker (7 of 7)** appears, which will show the output from the last node in the workflow.<br />
   f. To explore the chunker's results, search for the text `"CompositeElement"` (including the quotation marks). Notice that the lengths of some of the chunks that immediately
   precede page breaks might be shortened due to the presence of the page break impacting the chunk's size.<br />

9. Optionally, you can try running this workflow again with the **Chunk by Similarity** strategy, as follows:

   a. Click the close (**X**) button above the output on the right side of the screen.<br />
   b. In the workflow designer, click the **Chunker** node and then, in the node's settings pane's **Details** tab, select **Chunk by Similarity**.<br />
   c. Under **Chunk by Similarity**, specify the following settings:

   * Check the box labelled **Include Original Elements**.
   * Set **Max Characters** to **500**.
   * Set **Similarity Threshold** to **0.99**.
   * Leave **Contextual Chunking** turned off.

   <Tip>
     *What does **Similarity Threshold** mean?*

     * The **Similarity Threshold** is a number between 0 and 1 exclusive (**0.01** to **0.99** inclusive).
     * **0.01** means that any two segments of text that are being compared to each other and are considered least identical in semantic meaning to each other are more likely to be combined into the same chunk together, when such combining must occur.
     * **0.99** means that any two segments of text that are being compared to each other and are considered almost identical in semantic meaning to each other are more likely to be combined into the same chunk together, when such combining must occur.
     * Numbers toward **0.01** bias toward least-identical semantic matches, while numbers toward **0.99** bias toward near-identical semantic matches.
   </Tip>

   d. Click **Test**.<br />
   e. In the **Test output** pane, make sure that **Chunker (7 of 7)** is showing. If not, click the right arrow (**>**) until **Chunker (7 of 7)** appears, which will show the output from the last node in the workflow.<br />
   f. To explore the chunker's results, search for the text `"CompositeElement"` (including the quotation marks). Notice that the lengths of many of the chunks fall well short of the **Max Characters** limit. This is because a similarity threshold
   of **0.99** means that only sentences or text segments with a near-perfect semantic match will be grouped together into the same chunk. This is an extremely high threshold, resulting in very short, highly specific chunks of text. <br />
   g. If you change **Similarity Threshold** to **0.01** and run the workflow again, searching for the text `"CompositeElement"` (including the quotation marks), many of the chunks will now come closer to the **Max Characters** limit. This is because a similarity threshold
   of **0.01** provides an extreme tolerance of differences between pieces of text, grouping almost anything together.<br />

10. When you are done, be sure to click the close (**X**) button above the output on the right side of the screen, to return to
    the workflow designer for the next step.

## Step 6 (Optional): Experiment with embedding

In this step, you generate [embeddings](/concepts/embedding) for your workflow. Embeddings are vectors of numbers that represent various aspects of the text that is extracted by Unstructured.
These vectors are stored or "embedded" next to the text itself in a vector store or vector database. Chatbots, agents, and other AI solutions can use
these vector embeddings to more efficiently and effectively find, analyze, and use the associated text. These vector embeddings are generated by an
embedding model that is provided by an embedding provider. For the best embedding model to apply to your use case, see the documentation for your target downstream application toolsets.

1. With the workflow designer active from the previous step, just before the **Destination** node, click the add (**+**) icon, and then click **Transform > Embedder**.

   <img alt="Adding an embedder node" />

2. In the node's settings pane's **Details** tab, under **Select Embedding Model**, for **Azure OpenAI**, select **Text Embedding 3 Small \[dim 1536]**.

   <Note>
     The list of available embedding models is constantly being updated. Your list might also be different, depending on your Unstructured
     account type. If **Azure OpenAI** and **Text Embedding 3 Small \[dim 1536]** is not available, choose another available model from the list. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

     If you have an Unstructured **Business** account and want to add more models to this list, contact your
     Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
     [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. With the "Chinese Characters" PDF file still selected in the **Source** node, click **Test**.

4. In the **Test output** pane, make sure that **Embedder (8 of 8)** is showing. If not, click the right arrow (**>**) until **Embedder (8 of 8)** appears, which will show the output from the last node in the workflow.

5. To explore the embeddings, search for the text `"embeddings"` (including the quotation marks).

   <Tip>
     *What do all of these numbers mean?*

     All by themselves, the numbers in the `embeddings` field of the output have no human-interpretable meaning on their own.
     However, when combined with the specific text that these numbers are associated with, and the embedding model's
     logic that was used to generate these numbers, the numbers in the `embeddings` field are extremely powerful when leveraged by
     downstream chatbots, agents, and other AI solutions.

     These numbers typically represent complex, abstract attributes about the text that are known only to the embedding model that generated these numbers. These
     attributes can be about the text's overall sentiment, intent, subject, semantic meaning, grammatical function, relationships between words, or any number of other things that the model is good at figuring out. This
     is why the embedding model you choose here must be the exact same embedding model that you use in any related chatbot, agent, or other
     AI solution that relies on these numbers. Otherwise, the numbers that are generated here will not have the same meaning
     downstream as well. Also, the number of *dimensions* (or the number of numbers in the `embeddings` field)
     you choose here must also be the exact same number of dimensions downstream as well.

     To repeat, the name and number of dimensions
     for the embedding model you choose here must be the exact same name and number of dimensions for the embedding model you use in your
     related downstream chatbots, agents, and other AI solutions that rely on this particular text and its associated embeddings that were generated here.
   </Tip>

6. When you are done, be sure to click the close (**X**) button above the output on the right side of the screen, to return to
   the workflow designer so that you can continue designing things later as you see fit.

## Step 7: Experiment with structured data extraction

In this step, you apply custom [structured data extraction](/concepts/structured-data-extractor/data-extractor) to your workflow. Structured data extraction is the process where Unstructured
automatically extracts the data from your source documents into a format that you define up front. For example, in addition to Unstructured
partitioning your source documents into elements with types such as `NarrativeText`, `UncategorizedText`, and so on, you can have Unstructured
output key information from the source documents in a custom structured data format, within a `DocumentData` element containing aJSON object with custom fields such as `name`, `address`, `phone`, `email`, and so on.

1. With the workflow designer active from the previous step, just before the **Destination** node, click the add (**+**) icon, and then click **Enrich > Extract**.

   <img alt="Adding an extract node" />

2. In the node's settings pane's **Details** tab, under **Provider**, select **Anthropic**. Under **Model**, select **Claude Sonnet 4.5**. This is the model that Unstructured will use to do the structured data extraction.

   <Note>
     The list of available models for structured data extraction is constantly being updated. Your list might also be different, depending on your Unstructured
     account type. If **Anthropic** and **Claude Sonnet 4.5** is not available, choose another available model from the list. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

     If you have an Unstructured **Business** account and want to add more models to this list, contact your
     Unstructured account administrator or Unstructured sales representative, or email Unstructured Support at
     [support@unstructured.io](mailto:support@unstructured.io).
   </Note>

3. Click **Upload JSON**.

4. in the **JSON Schema** box, enter the following JSON schema, and then click **Use this Schema**:

   ```json theme={null}
   {
     "type": "object",
     "properties": {
       "title": {
         "type": "string",
         "description": "Full title of the research paper"
       },
       "authors": {
         "type": "array",
         "items": {
           "type": "object",
           "properties": {
             "name": {
               "type": "string",
               "description": "Author's full name"
             },
             "affiliation": {
               "type": "string",
               "description": "Author's institutional affiliation"
             },
             "email": {
               "type": "string",
               "description": "Author's email address"
             }
           },
           "required": [
             "name",
             "affiliation",
             "email"
           ],
           "additionalProperties": false
         },
         "description": "List of paper authors with their affiliations"
       },
       "abstract": {
         "type": "string",
         "description": "Paper abstract summarizing the research"
       },
       "introduction": {
         "type": "string",
         "description": "Introduction section describing the problem and motivation"
       },
       "methodology": {
         "type": "object",
         "properties": {
           "approach_name": {
             "type": "string",
             "description": "Name of the proposed method (e.g., StrokeNet)"
           },
           "description": {
             "type": "string",
             "description": "Detailed description of the methodology"
           },
           "key_techniques": {
             "type": "array",
             "items": {
               "type": "string"
             },
             "description": "List of key techniques used in the approach"
           }
         },
         "required": [
           "approach_name",
           "description",
           "key_techniques"
         ],
         "additionalProperties": false
       },
       "experiments": {
         "type": "object",
         "properties": {
           "datasets": {
             "type": "array",
             "items": {
               "type": "object",
               "properties": {
                 "name": {
                   "type": "string",
                   "description": "Dataset name"
                 },
                 "description": {
                   "type": "string",
                   "description": "Dataset description"
                 },
                 "size": {
                   "type": "string",
                   "description": "Dataset size (e.g., number of sentence pairs)"
                 }
               },
               "required": [
                 "name",
                 "description",
                 "size"
               ],
               "additionalProperties": false
             },
             "description": "Datasets used for evaluation"
           },
           "baselines": {
             "type": "array",
             "items": {
               "type": "string"
             },
             "description": "Baseline methods compared against"
           },
           "evaluation_metrics": {
             "type": "array",
             "items": {
               "type": "string"
             },
             "description": "Metrics used for evaluation"
           },
           "experimental_setup": {
             "type": "string",
             "description": "Description of experimental configuration and hyperparameters"
           }
         },
         "required": [
           "datasets",
           "baselines",
           "evaluation_metrics",
           "experimental_setup"
         ],
         "additionalProperties": false
       },
       "results": {
         "type": "object",
         "properties": {
           "main_findings": {
             "type": "string",
             "description": "Summary of main experimental findings"
           },
           "performance_improvements": {
             "type": "array",
             "items": {
               "type": "object",
               "properties": {
                 "dataset": {
                   "type": "string",
                   "description": "Dataset name"
                 },
                 "metric": {
                   "type": "string",
                   "description": "Evaluation metric (e.g., BLEU)"
                 },
                 "baseline_score": {
                   "type": "number",
                   "description": "Baseline method score"
                 },
                 "proposed_score": {
                   "type": "number",
                   "description": "Proposed method score"
                 },
                 "improvement": {
                   "type": "number",
                   "description": "Improvement over baseline"
                 }
               },
               "required": [
                 "dataset",
                 "metric",
                 "baseline_score",
                 "proposed_score",
                 "improvement"
               ],
               "additionalProperties": false
             },
             "description": "Performance improvements over baselines"
           },
           "parameter_reduction": {
             "type": "string",
             "description": "Description of parameter reduction achieved"
           }
         },
         "required": [
           "main_findings",
           "performance_improvements",
           "parameter_reduction"
         ],
         "additionalProperties": false
       },
       "related_work": {
         "type": "string",
         "description": "Summary of related work and prior research"
       },
       "conclusion": {
         "type": "string",
         "description": "Conclusion section summarizing contributions and findings"
       },
       "limitations": {
         "type": "string",
         "description": "Limitations and challenges discussed in the paper"
       },
       "acknowledgments": {
         "type": "string",
         "description": "Acknowledgments section"
       },
       "references": {
         "type": "array",
         "items": {
           "type": "string"
         },
         "description": "List of cited references"
       }
     },
     "additionalProperties": false,
     "required": [
       "title",
       "authors",
       "abstract",
       "introduction",
       "methodology",
       "experiments",
       "results",
       "related_work",
       "conclusion",
       "limitations",
       "acknowledgments",
       "references"
     ]
   }
   ```

5. With the "Chinese Characters" PDF file still selected in the **Source** node, click **Test**.

6. In the **Test output** pane, make sure that **Extract (9 of 9)** is showing. If not, click the right arrow (**>**) until **Extract (9 of 9)** appears, which will show the output from the last node in the workflow.

7. To explore the structured data extraction, search for the text `"extracted_data"` (including the quotation marks).

8. When you are done, be sure to click the close (**X**) button above the output on the right side of the screen, to return to
   the workflow designer so that you can continue designing things later as you see fit.

## Next steps

Congratulations! You now have an Unstructured workflow that partitions, enriches, chunks, embeds, and extracts structured data from your source documents, producing
context-rich data that is ready for retrieval-augmented generation (RAG), agentic AI, and model fine-tuning.

Right now, your workflow only accepts one local file at a time for input. Your workflow also only sends Unstructured's processed data to your screen or to be saved locally as a JSON file.
You can modify your workflow to accept multiple files and data from—and send Unstructured's processed data to—one or more file storage
locations, databases, and vector stores. To learn how to do this, try one or more of the following quickstarts:

* [Remote quickstart](/pipelines/quickstart#remote-quickstart) - This quickstart shows you how to begin processing files and semi-structured data from remote source locations at scale, instead of just one local file at a time.
* [Dropbox source connector quickstart](/pipelines/sources/dropbox-source-quickstart) - If you don't have any remote source locations available for Unstructured to connect to, this quickstart shows you how to set up a Dropbox account to store your documents in, and then connect Unstructured to your Dropbox account.
* [Pinecone destination connector quickstart](/pipelines/destinations/pinecone-destination-quickstart) - If you don't have any remote destination locations available for Unstructured to send its processed data to, this quickstart shows you how to set up a Pinecone account to have Unstructured store its processed data in, and then connect Unstructured to your Pinecone account.

Unstructured also offers an API and SDKs, which allow you to use code to work with Unstructured programmatically instead of only with Pipelines. For details, see:

* [Unstructured API quickstart](/api-reference/workflow/overview#quickstart) - This quickstart uses the Unstructured API's workflow operations to programmatically create a Dropbox source connector and a Pinecone destination connector in your Unstructured account. You then programmatically add these connectors to a workflow in your Unstructured account, run that workflow as a job, and then explore the job's results.
* [Unstructured Python SDK](/api-reference/workflow/overview#unstructured-python-sdk) - This article provides an overview of the Unstructured Python SDK and how to use it.
* [Unstructured API overview](/api-reference/overview) - This article provides an overview of the Unstructured API and how to use it.

If you are not able to complete any of the preceding quickstarts, contact Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# Webhooks
Source: https://docs.unstructured.io/pipelines/webhooks



A *webhook* is a common technique that allows a *sender* to automatically send data to a *receiver* based on events that happen in the sender.
Some common uses for webhooks include sending real-time notifications such as emails, pages, or alerts to messaging apps; automatically
triggering build, test, or deployment workflows in CI/CD pipelines; launching downstream data synchronization updates; or triggering agentic AI workflows.

The following 3-minute video provides an overview of webhooks.

<iframe title="YouTube video player" />

The following 4-minute video shows how to use Unstructured Pipelines to work with webhooks:

<iframe title="YouTube video player" />

For Unstructured webhooks, Unstructured is the sender,
and your solution is the receiver. Some popular receiver solutions include
[AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/urls-webhook-tutorial.html),
[Azure Functions](https://learn.microsoft.com/azure/azure-functions/functions-bindings-http-webhook),
[Google Cloud Run](https://docs.cloud.google.com/run/docs/triggering/webhooks),
[Zapier](https://zapier.com/blog/what-are-webhooks/),
[Slack](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/),
[Svix](https://svix.com), and
[Webhook.site](https://webhook.site/).

Your solution provides a unique URL, called the
*webhook URL*, to receive the data that Unstructured sends.
Behind the scenes, when specific events happen in Unstructured, Unstructured automatically calls the webhook URL and
sends JSON payloads related to those events. The receiver then processes the payloads and decides what to do with the data.

Because webhooks are event-driven, some event must first be triggered to begin generating the related data to be sent. In
Unstructured, these webhooks can be triggered whenever a [job](/pipelines/jobs) that is associated with its target [workflow](/pipelines/workflows) does
one or more of the following:

* Is scheduled to start running later (in the user interface, **Job scheduled**)
* Has begun running  (**Job in progress**)
* Has stopped running (**Job stopped**)
* Has failed to initialize  (`job.failed`) -  the job did not process any files.
* Has completed (`job.completed`) - the job initialized and finished processing. To determine whether all files were processed successfully, get the job's processing details. For more information, see [Get processing details for a job](/api-reference/api/job/get-job-details).

You can use Unstructured Pipelines to create and manage webhooks in your Unstructured personal account, or for a
[workspace](/pipelines/account/workspaces) within an Unstructured **Business** account. This allows any job that is associated with any workflow across this scope
to trigger the webhook. This can be useful, for example, for routing pager requests to a team of on-call engineers in an IT operations center
whenever any job fails within the given scope.

When you create a webhook in Pipelines, you can optionally provide a signing secret to [verify incoming webhook requests](/api-reference/webhooks#verify-webhook-requests).  If you do not provide a secret, you cannot use secret-based verification for incoming webhook requests. The secret is write-only and is not returned by the API after creation.

## View your personal account webhooks

To view the webhooks for your Unstructured personal account, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. If you have an Unstructured **Business** account, in the top navigation bar's account selector, select **Personal Account**.
3. In the sidebar, click your user icon, and then click **Account Settings**.
4. In the top navigation bar, click the **Webhooks** tab.

The webhooks for your personal account are displayed.

## View workspace webhooks

To view the webhooks for a workspace within an Unstructured **Business** account, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. In the top navigation bar, in the account selector, select the account for your target workspace.
3. Next to the account selector, in the workspace selector, select your target workspace.
4. In the sidebar, click the **Settings** (gear) icon, and then click **Manage Workspace**.
5. Below the top navigation bar, click the **Webhooks** tab.

The webhooks for this workspace are displayed.

## Create a webhook

To create a webhook that sends notifications for all specified events within the selected scope, regardless of the
resource that triggers the event, do the following:

1. If you are not already signed in, sign in to your Unstructured account.

2. [View your personal account webhooks](#view-your-personal-account-webhooks) or
   [view workspace webhooks](#view-workspace-webhooks), depending on the scope of the webhook you want to create.

3. Click **Add Webhook +**.

4. In the **Webhook Setup** dialog, for **Webhook URL**, enter the target receiver's webhook URL.

5. Optionally, for **Description**, enter a meaningful description of the webhook, for your own reference.

6. Optionally, for **Secret**, enter a strong, random signing secret in plain text. Use this secret to [verify incoming webhook requests](/api-reference/webhooks#verify-webhook-requests). If you do not provide a secret, you cannot use secret-based verification for incoming webhook requests. The secret must be between 24 and 75 bytes (24 to 75 ASCII characters).

7. For **Select Events**, click one or more of the available events that you want Unstructured to send notifications for.
   For more information about the list of available events, see this article's introductory section.

8. Click **Save Webhook**.

The new webhook is created and appears in the list of available webhooks.

## Change a webhook's settings

To change the settings for an existing webhook, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. [View your personal account webhooks](#view-your-personal-account-webhooks) or
   [view workspace webhooks](#view-workspace-webhooks), depending on where the target webhook is located.
3. In the list of available webhooks, click the name of the webhook you want to change the settings for.
4. In the webhook's settings pane, make the desired changes.
5. When you are done making your changes, click **Save Changes**.

The webhook's settings are updated, and the new settings begin taking effect.

## Delete a webhook

<Warning>
  Deleting a webhook is a permanent action and is not recoverable.
</Warning>

To delete an existing webhook, do the following:

1. If you are not already signed in, sign in to your Unstructured account.
2. [View your personal account webhooks](#view-your-personal-account-webhooks) or
   [view workspace webhooks](#view-workspace-webhooks), depending on where the target webhook is located.
3. In the list of available webhooks, click the name of the webhook you want to delete.
4. In the webhook's settings pane, click **Delete Webhook**.

The webhook is deleted, and event data will no longer be sent through this webhook.

## Unstructured Pipelines and the API support for webhooks: *workspace* and *workflow*

Unstructured Pipelines supports creating and managing webhooks in your Unstructured personal account, or for a *workspace* within an Unstructured **Business** account.

This page describes *workspace-scoped* webhook actions that you can perform in Pipelines. You can also use the [Unstructured API](/api-reference/webhooks) to perform these same actions programmatically. In addition, the API supports *workflow-scoped* webhooks.

The API also offers the following:

* Create and manage webhooks at the workflow level, known as a *workflow-scoped notification channel*. This allows any job that is associated only with the target workflow to trigger the webhook. This can be useful, for example, for emailing a department's data analyst when a long-running job for a specific workflow has completed, allowing them to begin working with the latest output.
* Manage notifications for both workspace and workflow. Whenever a webhook is triggered, the act of Unstructured sending the event data payload is called a *notification*. You can get a count or a list of these notifications. You can also mark them as read after you have processed them according to your organization's needs. Marking a notification as read can help you keep track of which notifications you have already dealt with and which ones you still need to take action on.
* Use the same secret-management flow described earlier for *workflow-scoped* webhooks, including optionally providing a secret at creation time and rotating it later through the update endpoint.

[Learn more](/api-reference/webhooks).


# Workflows
Source: https://docs.unstructured.io/pipelines/workflows



## Workflows dashboard

<img alt="Workflows in the sidebar" />

To view the workflows dashboard, on the sidebar, click **Workflows**.

A workflow in Unstructured is a defined sequence of processes that automate the data handling from source to destination. It allows users to configure how and when data should be ingested, processed, and stored.

Workflows are crucial for establishing a systematic approach to managing data flows within the platform, ensuring consistency, efficiency, and adherence to specific data processing requirements.

## Create a workflow

Unstructured provides two types of workflow builders:

* [Automatic](#create-an-automatic-workflow) or **Build it For Me** workflows, which use sensible default workflow settings to enable you to get good-quality results faster.
* [Custom](#create-a-custom-workflow) or **Build it Myself** workflows, which enable you to fine-tune the workflow settings behind the scenes to get very specific results.

### Create an automatic workflow

<Warning>
  You must first have an existing source connector and destination connector to add to the workflow.

  You cannot create an automatic workflow that uses a local file as a source.

  If you do not have an existing remote connector for either your target source (input) or destination (output) location, [create the source connector](/pipelines/sources/overview), [create the destination connector](/pipelines/destinations/overview), and then return here.

  To see your existing connectors, on the sidebar, click **Connectors**, and then click **Sources** or **Destinations**.
</Warning>

To create an automatic workflow:

1. On the sidebar, click **Workflows**.

2. Click **New Workflow**.

3. Next to **Build it for Me**, click **Create Workflow**.

   <Note>If a radio button appears instead of **Build it for Me**, select it, and then click **Continue**.</Note>

4. For **Workflow Name**, enter some unique name for this workflow.

5. In the **Sources** dropdown list, select your source location.

6. In the **Destinations** dropdown list, select your destination location.

   <Note>You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.</Note>

7. Click **Continue**.

8. Check or uncheck the **Reprocess All** box:

   * Checking this box reprocesses all documents in the source location on every workflow run.
   * Unchecking this box excludes from future processing any files Unstructured determines are unchanged since the last time the workflow ran.

   Unstructured determines if a document has changed based on the document version. For each workflow, Unstructured maintains a record of documents (and their versions, if present) processed by that workflow. Each document record consists of:

   * A record ID derived from the document name and path.
   * A record version derived from either the document Etag (if the source provider generates one) or the source provider's native version identifier.

   When you uncheck **Reprocess All** for a source connector that supports this option, Unstructured uses this list of records to determine whether or not to process each document:

   * If the record ID does not exist in the workflow records, Unstructured processes the document.
   * If the record ID exists, but the record version has changed, or there is no record version, Unstructured processes the document.

   The following table lists out the possible record ID and record version combinations, and the action Unstructured takes in each case:

   | Record ID | Record Version   | Action              |
   | --------- | ---------------- | ------------------- |
   | Exists    | Unchanged        | Do not process file |
   | Exists    | Changed          | Process file        |
   | Exists    | (none)           | Process file        |
   | New       | (Does not apply) | Process file        |

   <Note>
     Renaming a document results in a new record ID; Unstructured will then reprocess the renamed document when the workflow runs.
   </Note>

   The following table lists the source connectors that support the **Reprocess All** setting. The **Record version base** column specifies the versioning information Unstructured uses to generated the corresponding record version for each processed document.

   Source connectors that do not support **Reprocess All** reprocess every document in the source location each time the workflow runs.

   | Connector                                                                | Record version base |
   | ------------------------------------------------------------------------ | ------------------- |
   | [Amazon S3](/api-reference/workflow/sources/s3)                          | ETag                |
   | [Azure Blob Storage](/api-reference/workflow/sources/azure-blob-storage) | ETag                |
   | [Box](/api-reference/workflow/sources/box)                               | Provider version ID |
   | [Dropbox](/api-reference/workflow/sources/dropbox)                       | Provider version ID |
   | [Elastisearch](/api-reference/workflow/sources/elasticsearch)            | Provider version ID |
   | [Google Cloud Storage](/api-reference/workflow/sources/google-cloud)     | ETag                |
   | [Google Drive](/api-reference/workflow/sources/google-drive)             | Provider version ID |
   | [Microsoft OneDrive](/api-reference/workflow/sources/onedrive)           | Provider version ID |
   | [Microsoft SharePoint](/api-reference/workflow/sources/sharepoint)       | Provider version ID |

   Additional considerations to take into account when unchecking **Reprocess All**:

   * Unstructured only adds document records for documents that it successfully processes. Documents that failed to process will be reprocessed the next time the workflow is run.
   * Because S3 ETags are content-based, changing the metadata on an S3 object will not result in it being reprocessed.
   * For source providers that support the S3 protocol, be aware that deleting an object and then reuploading it to the source location will maintain the same record ID, but may result in a different record version being generated. This is especially true of multipart uploads. This results in Unstructured reprocessing the document.
   * For source providers that offer Key Management Services (KMS), be aware that server-side encryption can change document ETags. This results in the the record version of a document changing, and Unstructured reprocessing the document.
   * If you clone or recreate a source connector, the resulting connector does not include the document processing history of the previous connector.
   * Changing a workflow's configuration does not automatically result in Unstructured reprocessing all documents. For example, changing chunker, embedder, enrichment, or partitioner settings may not result in reprocessing all document. To reprocess all documents using new workflow settings, check **Reprocess All** for at least the next workflow run.

9. Click **Continue**.

10. If you want this workflow to run on a schedule, in the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select **Don't repeat**.

11. Click **Complete**.

By default, this workflow partitions, chunks, and generates embeddings as follows:

* **Partitioner**: **Auto** strategy

  Unstructured automatically analyzes and processes files on a page-by-page basis (for PDF files) and on a document-by-document basis for everything else:

  * If the page or document has no images and likely does not have tables, **Fast** partitioning is used, and the page or document is billed at the **Fast** rate for processing.
  * If the page or document has only a few tables or images with standard layouts and languages, **High Res** partitioning is used, and the page or document is billed at the **High Res** rate for processing.
  * If the page or document has more than a few tables or images, **VLM** partitioning is used, and the page or document is billed at the **VLM** rate for processing.

  [Learn about partitioning strategies](/concepts/partitioning).

* **Chunker**: **Chunk by Title** strategy

  * **Contextual Chunking**: No (unchecked)
  * **Combine Text Under N Characters**: 3000
  * **Include Original Elements**: Yes (checked)
  * **Max Characters**: 5500
  * **Multipage Sections**: Yes (checked)
  * **New After N Characters**: 3500
  * **Overlap**: 350
  * **Overlap All**: Yes (checked)

  [Learn about chunking strategies](/concepts/chunking).

* **Embedder**:

  * **Provider**: Azure OpenAI
  * **Model**: text-embedding-3-large, with 3072 dimensions

  [Learn about embedding providers and models](/concepts/embedding).

* Enrichments:

  This workflow contains no enrichments, other than a **Chunker** node.

  [Learn about available enrichments](/concepts/enriching/overview).

After this workflow is created, you can change any or all of its settings if you want to. This includes the workflow's
source connector, destination connector, partitioning, chunking, and embedding settings. You can also add enrichments
to the workflow if you want to.

<Warning>
  Unstructured can potentially generate image summary descriptions, table summary descriptions, table-to-HTML output, and generative OCR optimizations, only for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is added.
  * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is needed (or allowed).

  Even with these configurations, Unstructured actually generates image summary descriptions, table summary descriptions, and table-to-HTML output only for files that contain images or tables and are also eligible
  for processing with the following partitioning strategies:

  * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
  * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

  Unstructured never generates image summary descriptions, table summary descriptions, or table-to-HTML output for workflows that are configured as follows:

  * With a **Partitioner** node set to use the **Fast** partitioning strategy.
  * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images or tables.

  Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
</Warning>

To change the workflow's default settings or to add enrichments:

1. On the sidebar, click **Workflows**.
2. In the list of available workflows, click the workflow that was just created. This opens a visual designer that shows
   your workflow as a directed acyclic graph (DAG). This DAG contains a node representing each step in the workflow.
   There is one node for the partitioning step, another node for the chunking step, and so on.
3. To learn how to change a node's settings or to add enrichment nodes, click the **FAQ** button in the flyout pane in
   the workflow DAG designer.

If you did not previously set the workflow to run on a schedule, you can [run the workflow](#edit-delete-or-run-a-workflow) now.

### Create a custom workflow

<Tip>
  If you already have an existing workflow that you want to change, do the following:

  1. On the sidebar, click **Workflows**.
  2. Click the name of the workflow that you want to change.
  3. Skip ahead to Step 11 in the following procedure.
</Tip>

<Warning>
  You can create and save a custom workflow that uses a local file as a source or does not have a source or destination connector added. However, you cannot activate the workflow or run the workflow
  either manually or on a schedule until a source and destination connector are added to the workflow.

  If you do not have an existing connector for either your target source or destination location, [create the source connector](/pipelines/sources/overview), [create the destination connector](/pipelines/destinations/overview), and then return here.

  To see your existing connectors, on the sidebar, click **Connectors**, and then click **Sources** or **Destinations**.
</Warning>

1. On the sidebar, click **Workflows**.

2. Click **New Workflow**.

3. Click the **Build it Myself** option, and then click **Continue**.

4. In the **This workflow** pane, click the **Details** button.

   <img alt="Workflow details" />

5. Next to **Name**, click the pencil icon, enter some unique name for this workflow, and then click the check mark icon.

6. If you want this workflow to run on a schedule, click the **Schedule** button. In the **Repeat Run** dropdown list, select one of the scheduling options, and fill in the scheduling settings.

7. To overwrite any previously processed files, or to retry any documents that fail to process, click the **Settings** button, and check either or both of the boxes.

   Check or uncheck the **Reprocess All** box:

   * Checking this box reprocesses all documents in the source location on every workflow run.
   * Unchecking this box excludes from future processing any files Unstructured determines are unchanged since the last time the workflow ran.

   Unstructured determines if a document has changed based on the document version. For each workflow, Unstructured maintains a record of documents (and their versions, if present) processed by that workflow. Each document record consists of:

   * A record ID derived from the document name and path.
   * A record version derived from either the document Etag (if the source provider generates one) or the source provider's native version identifier.

   When you uncheck **Reprocess All** for a source connector that supports this option, Unstructured uses this list of records to determine whether or not to process each document:

   * If the record ID does not exist in the workflow records, Unstructured processes the document.
   * If the record ID exists, but the record version has changed, or there is no record version, Unstructured processes the document.

   The following table lists out the possible record ID and record version combinations, and the action Unstructured takes in each case:

   | Record ID | Record Version   | Action              |
   | --------- | ---------------- | ------------------- |
   | Exists    | Unchanged        | Do not process file |
   | Exists    | Changed          | Process file        |
   | Exists    | (none)           | Process file        |
   | New       | (Does not apply) | Process file        |

   <Note>
     Renaming a document results in a new record ID; Unstructured will then reprocess the renamed document when the workflow runs.
   </Note>

   The following table lists the source connectors that support the **Reprocess All** setting. The **Record version base** column specifies the versioning information Unstructured uses to generated the corresponding record version for each processed document.

   Source connectors that do not support **Reprocess All** reprocess every document in the source location each time the workflow runs.

   | Connector                                                                | Record version base |
   | ------------------------------------------------------------------------ | ------------------- |
   | [Amazon S3](/api-reference/workflow/sources/s3)                          | ETag                |
   | [Azure Blob Storage](/api-reference/workflow/sources/azure-blob-storage) | ETag                |
   | [Box](/api-reference/workflow/sources/box)                               | Provider version ID |
   | [Dropbox](/api-reference/workflow/sources/dropbox)                       | Provider version ID |
   | [Elastisearch](/api-reference/workflow/sources/elasticsearch)            | Provider version ID |
   | [Google Cloud Storage](/api-reference/workflow/sources/google-cloud)     | ETag                |
   | [Google Drive](/api-reference/workflow/sources/google-drive)             | Provider version ID |
   | [Microsoft OneDrive](/api-reference/workflow/sources/onedrive)           | Provider version ID |
   | [Microsoft SharePoint](/api-reference/workflow/sources/sharepoint)       | Provider version ID |

   Additional considerations to take into account when unchecking **Reprocess All**:

   * Unstructured only adds document records for documents that it successfully processes. Documents that failed to process will be reprocessed the next time the workflow is run.
   * Because S3 ETags are content-based, changing the metadata on an S3 object will not result in it being reprocessed.
   * For source providers that support the S3 protocol, be aware that deleting an object and then reuploading it to the source location will maintain the same record ID, but may result in a different record version being generated. This is especially true of multipart uploads. This results in Unstructured reprocessing the document.
   * For source providers that offer Key Management Services (KMS), be aware that server-side encryption can change document ETags. This results in the the record version of a document changing, and Unstructured reprocessing the document.
   * If you clone or recreate a source connector, the resulting connector does not include the document processing history of the previous connector.
   * Changing a workflow's configuration does not automatically result in Unstructured reprocessing all documents. For example, changing chunker, embedder, enrichment, or partitioner settings may not result in reprocessing all document. To reprocess all documents using new workflow settings, check **Reprocess All** for at least the next workflow run.

8. The workflow begins with the following layout:

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Destination
   ```

   The following workflow layouts are also valid:

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Chunker-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Chunker-->Embedder-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Enrichment-->Chunker-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Enrichment-->Chunker-->Embedder-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Extract-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Chunker-->Extract-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Chunker-->Embedder-->Extract-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Enrichment-->Chunker-->Extract-->Destination
   ```

   ```mermaid theme={null}
   flowchart LR
     Source-->Partitioner-->Enrichment-->Chunker-->Embedder-->Extract-->Destination
   ```

   <Note>
     For workflows that use **Chunker** and enrichment nodes together, the **Chunker** node should be placed after all enrichment nodes. Placing the
     **Chunker** node before any enrichment nodes could cause incomplete or no enrichment results to be generated.
   </Note>

   <Warning>
     You can create and save a workflow that does not use a valid workflow layout. However, you cannot activate the workflow or run the workflow
     either manually or on a schedule until the workflow is changed to use a valid workflow layout.
   </Warning>

9. In the pipeline designer, click the **Source** node. In the **Source** pane, select the source location. Then click **Save**.

   <img alt="Workflow designer" />

   <Note>
     To use a local source location, do not choose a source connector.

     If the workflow uses a local source location, in the **Source** node, drag or click to specify a local file, and then click **Test**. The workflow's
     results are displayed on-screen.

     A workflow that uses a local source location has the following limitations:

     * You cannot save the workflow.
     * You cannot send the results to a remote destination location, even if you have attached a destination connector to
       the workflow. However, you can save the results to a local JSON-formatted file.
   </Note>

10. Click the **Destination** node. In the **Destination** pane, select the destination location. Then click **Save**.

11. As needed, add more nodes by clicking the plus icon (recommended) or **Add Node** button:

    <img alt="Add node to workflow" />

    * Click **Connect** to add another **Source** or **Destination** node. You can add multiple source and destination locations. Files will be ingested from all of the source locations, and the processed data will be delivered to all of the destination locations. [Learn more](#custom-workflow-node-types).

    * Click **Enrich** to add a chunker or enrichment node. [Learn more](#custom-workflow-node-types).

      <Warning>
        Unstructured can potentially generate image summary descriptions, table summary descriptions, table-to-HTML output, and generative OCR optimizations, only for workflows that are configured as follows:

        * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is added.
        * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is needed (or allowed).

        Even with these configurations, Unstructured actually generates image summary descriptions, table summary descriptions, and table-to-HTML output only for files that contain images or tables and are also eligible
        for processing with the following partitioning strategies:

        * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
        * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

        Unstructured never generates image summary descriptions, table summary descriptions, or table-to-HTML output for workflows that are configured as follows:

        * With a **Partitioner** node set to use the **Fast** partitioning strategy.
        * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images or tables.

        Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
      </Warning>

    * Click **Transform** to add a **Partitioner** or **Embedder** node. [Learn more](#custom-workflow-node-types).

      <Warning>
        If you add an **Embedder** node, you must set the **Chunker** node's **Max Characters** setting to a value at or below Unstructured's recommended
        maximum chunk size for your selected embedding model. [Learn more](/concepts/embedding#chunk-sizing-and-embedding-models).
      </Warning>

    <Tip>
      Make sure to add nodes in the correct order. If you are unsure, see the usage hints in the blue note that appears
      in the node's settings pane.

      <img alt="Node usage hints note" />
    </Tip>

    To edit a node, click that node, and then change its settings.

    To delete a node, click that node, and then click the trash can icon above it.

12. Click **Save**.

13. If you did not set the workflow to run on a schedule, you can [run the workflow](#edit-delete-or-run-a-workflow) now.

#### Custom workflow node types

<AccordionGroup>
  <Accordion title="Partitioner node">
    Choose from one of four available partitioning strategies.

    Unstructured recommends that you choose the **Auto** partitioning strategy in most cases. With **Auto**, Unstructured does all
    the heavy lifting, optimizing at runtime for the highest quality at the lowest cost page-by-page.

    You should consider the following additional strategies only if you are absolutely sure that your documents are of the same
    type. Each of the following strategies are best suited for specific situations. Choosing one of these
    strategies other than **Auto** for sets of documents of different types could produce undesirable results,
    including reduction in transformation quality.

    * **VLM**: For the highest-quality transformation of these file types: `.bmp`, `.gif`, `.heic`, `.jpeg`, `.jpg`, `.pdf`, `.png`, `.tiff`, and `.webp`.
    * **High Res**: For all other [supported file types](/pipelines/supported-file-types), and for the generation of bounding box coordinates.
    * **Fast**: For text-only documents.

    The **Auto** partitioning strategy routes each file as a complete unit to the appropriate partitioning strategy (**VLM**, **High Res**, or **Fast**)
    based on the preceding file types. Additionally, for `.pdf` files, the **Auto** partitioning strategy routes these files' pages
    on a page-by-page basis, as follows:

    * A page is routed to **Fast** when it contains only embedded text and no images or tables are detected.
    * All other kinds of pages are routed to **VLM** or **High Res**, depending on the complexity of a page's
      content. Unstructured constantly optimizes its proprietary algorithm for routing to **VLM** or **High Res** in these cases.

    For **VLM**, you must also choose a VLM provider and model from among the available choices that are shown. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

    <Warning>
      The following models are no longer available as of the following dates:

      * Amazon Bedrock Claude Sonnet 3.5: October 22, 2025
      * Anthropic Claude Sonnet 3.5: October 22, 2025

      Unstructured recommends the following actions:

      * For new workflows, do not use any of these models.
      * For any workflow that uses any of these models, update that workflow as soon as possible to use a different model.

      Workflows that attempt to use any of these models on or after its associated date will return errors.
    </Warning>

    <Note>
      When you use the **VLM** strategy with embeddings for PDF files of 200 or more pages, you might notice some errors when
      these files are processed. These errors typically occur when these larger PDF files have lots of tables and high-resolution images.
    </Note>

    If you choose the **Fast** strategy, you can also choose from among the following additional settings:

    * **Include Page breaks**: Check this box to include distinct `PageBreak` document elements in the output, if the file type supports it.
    * **Infer Table Structure**: Check this box to add, for each table in a PDF file, a metadata field named `text_as_html` to the output for that table's document element. This field will contain an HTML representation of the table.
    * **Elements to Exclude**: Select the name of each available type of [document element](/concepts/document-elements) to exclude from the output.

    If you choose the **High Res** strategy, you can also choose from among the following additional settings:

    * **Include Page breaks**: Check this box to include distinct `PageBreak` document elements in the output, if the file type supports it.
    * **Infer Table Structure**: Check this box to add, for each table in a PDF file, a metadata field named `text_as_html` to the output for that table's document element. This field will contain an HTML representation of the table.
    * **Include Coordinates**: Check this box to add, for each [document element](/concepts/document-elements) in the output, a metadata field named `coordinates` to the output for that document element. This field will contain the bounding box coordinates of the document element's content on the page, as well as the bounding box's width and height in pixels.
    * **Extract Image Block Types**: Select the name of each available type of document element to add a metadata field named `image_base64` to the output for that document element. This field will contain a Base64-encoded representation of the document element's content. A Base64-to-image decoding of this field's value will return an image representing the document element's original content.
    * **Elements to Exclude**: Select the name of each available type of document element to exclude from the output.

    [Learn more](/concepts/partitioning).
  </Accordion>

  <Accordion title="Chunker node">
    For **Chunkers**, select one of the following:

    * **Chunk by title**: Preserve section boundaries and optionally page boundaries as well. A single chunk will never contain text that occurred in two different sections. When a new section starts, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk. Also, specify the following:

      * **Contextual chunking**: When switched on, prepends chunk-specific explanatory context to each chunk. [Learn more](/concepts/chunking#contextual-chunking).
      * **Combine text under n chars**: Combine elements until a section reaches a length of this many characters. The default is **0**.
      * **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
      * **Max characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **2048**.
      * **Multipage sections**: Check this box to allow sections to span multiple pages. By default, this box is unchecked.
      * **New after n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is **1500**.
      * **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **160**.
      * **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    * **Chunk by character** (also known as *basic* chunking): Combine sequential elements to maximally fill each chunk. Also, specify the following:

      * **Contextual chunking**: When switched on, prepends chunk-specific explanatory context to each chunk. [Learn more](/concepts/chunking#contextual-chunking).
      * **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
      * **Max characters**: Cut off new sections after reaching a length of this many characters. The default is **2048**.
      * **New after n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is **1500**.
      * **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **160**.
      * **Overlap All**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    * **Chunk by page**: Preserve page boundaries. When a new page is detected, the existing chunk is closed and a new one is started, even if the next element would fit in the prior chunk. Also, specify the following:

      * **Contextual chunking**: When switched on, prepends chunk-specific explanatory context to each chunk. [Learn more](/concepts/chunking#contextual-chunking).
      * **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
      * **Max characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **500**.
      * **New after n chars**: Cut off new sections after reaching a length of this many characters. This is an approximate limit. The default is **50**.
      * **Overlap**: Apply a prefix of this many trailing characters from the prior text-split chunk to second and later chunks formed from oversized elements by text-splitting. The default is **30**.
      * **Overlap all**: Check this box to apply overlap to "normal" chunks formed by combining whole elements. Use with caution as this can introduce noise into otherwise clean semantic units. By default, this box is unchecked.

    * **Chunk by similarity**: Use the [sentence-transformers/multi-qa-mpnet-base-dot-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1) embedding model to identify topically similar sequential elements and combine them into chunks. Also, specify the following:

      * **Contextual chunking**: When switched on, prepends chunk-specific explanatory context to each chunk. [Learn more](/concepts/chunking#contextual-chunking).
      * **Include original elements**: Check this box to output the elements that were used to form a chunk, to appear in the `metadata` field's `orig_elements` field for that chunk. By default, this box is unchecked.
      * **Max characters**: Cut off new sections after reaching a length of this many characters. This is a strict limit. The default is **500**.
      * **Similarity threshold**: Specify a threshold between 0 and 1 exclusive (0.01 to 0.99 inclusive), where 0 indicates completely dissimilar vectors and 1 indicates identical vectors, taking into consideration the trade-offs between precision (a higher threshold) and recall (a lower threshold). The default is **0.5**. [Learn more](https://towardsdatascience.com/introduction-to-embedding-clustering-and-similarity-11dd80b00061).

    Learn more:

    * [Chunking overview](/concepts/chunking)
    * [Chunking for RAG: best practices](https://unstructured.io/blog/chunking-for-rag-best-practices)
  </Accordion>

  <Accordion title="Enrichment node">
    Each enrichment node type has its own unique settings:

    <Warning>
      The following models are no longer available as of the following dates:

      * Amazon Bedrock Claude Sonnet 3.5: October 22, 2025
      * Anthropic Claude Sonnet 3.5: October 22, 2025

      Unstructured recommends the following actions:

      * For new workflows, do not use any of these models.
      * For any workflow that uses any of these models, update that workflow as soon as possible to use a different model.

      Workflows that attempt to use any of these models on or after its associated date will return errors.
    </Warning>

    For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

    <Warning>
      Unstructured can potentially generate image summary descriptions, table summary descriptions, table-to-HTML output, and generative OCR optimizations, only for workflows that are configured as follows:

      * With a **Partitioner** node set to use the **Auto** or **High Res** partitioning strategy, and an image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is added.
      * With a **Partitioner** node set to use the **VLM** partitioning strategy. No image summary description node, table summary description node, table-to-HTML output node, or generative OCR optimization node is needed (or allowed).

      Even with these configurations, Unstructured actually generates image summary descriptions, table summary descriptions, and table-to-HTML output only for files that contain images or tables and are also eligible
      for processing with the following partitioning strategies:

      * **High Res**, when the workflow's **Partitioner** node is set to use **Auto** or **High Res**.
      * **VLM** or **High Res**, when the workflow's **Partitioner** node is set to use **VLM**.

      Unstructured never generates image summary descriptions, table summary descriptions, or table-to-HTML output for workflows that are configured as follows:

      * With a **Partitioner** node set to use the **Fast** partitioning strategy.
      * With a **Partitioner** node set to use the **Auto**, **High Res**, or **VLM** partitioning strategy, for all files that Unstructured encounters that do not contain images or tables.

      Unstructured never produces generative OCR optimizations for workflows with a **Partitioner** node set to use the **Fast** partitioning strategy.
    </Warning>

    * The **Image Description** node summarizes images. You must select one of the available provider (and model) combinations that are shown.

      <Note>
        For workflows that use [chunking](/concepts/chunking), the **Chunker** node should be placed after all enrichment nodes. Placing the
        **Chunker** node before an image description enrichment node could cause incomplete or no image descriptions to be generated.
      </Note>

      [Learn more](/concepts/enriching/image-descriptions).

    * The **Table Description** node summarizes tables. You must select one of the available provider (and model) combinations that are shown.

      <Note>
        For workflows that use [chunking](/concepts/chunking), the **Chunker** node should be placed after all enrichment nodes. Placing the
        **Chunker** node before a table description enrichment node could cause incomplete or no table descriptions to be generated.
      </Note>

      [Learn more](/concepts/enriching/table-descriptions).

    * The **Table to HTML** node generates HTML representations for tables. Also select the following:

      * To use agentic AI to increase HTML accuracy for complex tables, select **Agentic** for **Mode**.
      * To use a VLM for standard tables, select **Standard** for **Mode**. Then select one of the available **Provider** and **Model** combinations that are shown.

      <Note>
        For workflows that use [chunking](/concepts/chunking), the **Chunker** node should be placed after all enrichment nodes. Placing the
        **Chunker** node before a table-to-HTML output enrichment node could cause incomplete or no table-to-HTML output to be generated.
      </Note>

      [Learn more](/concepts/enriching/table-to-html).

    * The **NER** node generates a list of recognized entities and their relationships by using a technique called *named entity recognition* (NER).
      You must select one of the available provider (and model) combinations that are shown.

      You can also customize the prompt used to add or remove entities and relationships. In the **Details** tab, under **Prompt**, click **Edit**. Click **Run Prompt** in the
      **Edit & Test Prompt** section to test the prompt.

      1. The selected model will follow a default set of instructions (called a *prompt*) to perform NER using a set of predefined entity types and relationships. To experiment
         with running the default prompt against some sample data, click **Edit**, and then click **Run Prompt**. The selected **Model** uses the
         **Prompt** to run NER on the **Input sample** and shows the results in the **Output**. Look specifically at the `response_json` field for the
         entities that were recognized and their relationships.

      2. To customize the prompt, change the contents of **Prompt**.

         <Note>
           For best results, Unstructured strongly recommends that you limit your changes only to certain portions of the default prompt, specifically:

           * Adding, renaming, or deleting items in the list of predefined types (such as `PERSON`, `ORGANIZATION`, `LOCATION`, and so on).

           * Adding, renaming, or deleting items in the list of predefined relationships (such as `works_for`, `based_in`, `has_role`, and so on).

           * As needed, adding any clarifying instructions only between these two lines:

             ```text theme={null}
             ...
             Provide the entities and their corresponding types as a structured JSON response.

             (Add any clarifying instructions here only.)

             [START OF TEXT]
             ...
             ```

           * Changing any other portions of the default prompt could produce unexpected results.
         </Note>

      3. To experiment with different data, change the contents of **Input sample**. For best results, Unstructured strongly recommends that the JSON structure in **Input sample** be preserved.

      4. When you are satisfied with the **Model** and **Prompt** that you want to use, click **Save**.

      [Learn more](/concepts/enriching/ner).

    * The **Generative OCR** node optimizes the fidelity of text blocks that Unstructured initially processed during its partitioning phase.
      You must select one of the available provider (and model) combinations that are shown.

      <Warning>
        Generative OCR does not process any text blocks by default. You must also explicitly specify which document element
        types containing text that you want generative OCR to process. To do this, in the workflow editor for your workflow:

        1. Click the **Partitioner** node.
        2. In the node's settings pane, scroll down to and then click a blank area inside of the **Extract Image Block Types** list.
        3. Select each [document element type](/concepts/document-elements#element-type) that you want generative OCR to process. For this
           walkthrough, select only **NarrativeText**.

        Generative OCR does not process the text of any `Image` or `Table` elements if they have already been processed by
        [image description](/api-reference/workflow/nodes/enrichment/enrichment-image-description) or [table description](/api-reference/workflow/nodes/enrichment/enrichment-table-description) enrichments, respectively. Do
        not remove the **Image** or **Table** document elements types from this **Extract Image Block Types** list, or else
        the image description and table description enrichments in your workflow might produce unexpected results or might not work at all.
      </Warning>

      <Note>
        For workflows that use [chunking](/concepts/chunking), the **Chunker** node should be placed after all enrichment nodes. Placing the **Chunker** node before an image descriptions enrichment node could cause incomplete or no image descriptions to be generated.
      </Note>

      [Learn more](/concepts/enriching/generative-ocr).
  </Accordion>

  <Accordion title="Embedder node">
    For **Select Embedding Model**, select one of the available models that are shown. For a full list of the models available in Unstructured, see [Available models](/api-reference/workflow/models).

    <Warning>
      If you add an **Embedder** node, you must set the **Chunker** node's **Max Characters** setting to a value at or below Unstructured's recommended
      maximum chunk size for your selected embedding model. [Learn more](/concepts/embedding#chunk-sizing-and-embedding-models).
    </Warning>

    Learn more:

    * [Embedding overview](/concepts/embedding)
    * [Understanding embedding models: make an informed choice for your RAG](https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag).
  </Accordion>

  <Accordion title="Extract node">
    Do one of the following to define the custom schema for the structured data that you want to extract:

    * To use a custom schema that conforms to the [OpenAI Structured Outputs](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas) guidelines,
      click **Upload JSON**; enter your own custom schema or upload a JSON file that contains your custom schema; and then click **Use this Schema**.
      [Learn about the OpenAI Structured Outputs format](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas).
    * To use a visual editor to define the schema, enter your own custom schema objects and their properties. To clear the current schema and start over,
      click the ellipses (three dots) icon, and then click **Reset form**.
      [Learn about OpenAI Structured Outputs data types](https://platform.openai.com/docs/guides/structured-outputs#supported-schemas).

    [Learn more](/concepts/structured-data-extractor/data-extractor).
  </Accordion>
</AccordionGroup>

## Edit, delete, or run a workflow

To run a workflow once, manually:

1. On the sidebar, click **Workflows**.
2. In the list of workflows, click **Run** in the row for the workflow that you want to run.

For each of the workflows on the **Workflows** list page, the following actions are available by clicking the ellipses (the three dots) in the row for the respective workflow:

* **Edit via Form**: Changes the existing configuration of your workflow.
* **Delete**: Removes the workflow from the platform. Use this action cautiously, as it will permanently delete the workflow and its configurations.
* **Open**: Opens the workflow's settings page.

## Pause a scheduled workflow

To stop running a workflow that is set to run on a repeating schedule:

1. On the sidebar, click **Workflows**.
2. In the list of workflows, turn off the **Status** toggle in the row for the workflow that you want to stop running on a repeated schedule.

Turning off the **Status** toggle also disables the workflow's **Run** button, which prevents that workflow from being run manually as well.

To resume running the workflow on its original repeating schedule, as well as enable the workflow to be run manually as needed, turn on the workflow's **Status** toggle.

## Duplicate a workflow

To duplicate (copy or clone) a workflow:

1. On the sidebar, click **Workflows**.
2. In the list of workflows, click the ellipses (the three dots) in the row for the workflow that you want to duplicate.
3. Click **Duplicate**.

   A duplicate of the workflow is created with the same configuration as the original workflow. The duplicate workflow has the same display name as the original
   workflow but with **(Copy)** at the end.


# Access to services
Source: https://docs.unstructured.io/support/access



On February 25, 2026, Unstructured stopped providing access to its services in specific locales, to
customers enrolled in Unstructured's **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** account plans.
Unstructured made this decision after careful review, determining that the compliance and regulatory
requirements associated with continuing to operate in these locales were no longer sustainable relative to
the locales' usage and revenue levels at the time of the decision.

You can continue to access Unstructured in the following locales:

* Austria
* Australia
* Belgium
* Bulgaria
* Canada
* Cyprus
* Czech Republic
* Germany
* Denmark
* Estonia
* Spain
* Finland
* France
* United Kingdom
* Greece
* Croatia
* Hungary
* Ireland
* India
* Italy
* Japan
* Republic of Korea
* Lithuania
* Luxembourg
* Latvia
* Malta
* Netherlands
* Norway
* Poland
* Portugal
* Romania
* Sweden
* Singapore
* Slovenia
* Slovakia
* United States of America

After February 25, 2026, organizations with large-scale needs located outside the previously listed locales may be able to continue accessing Unstructured through a
signed Unstructured **Business** account agreement, covering **Business SaaS**, **dedicated instance** , or **in-VPC** deployments
that addresses the applicable compliance and regulatory requirements. If you are interested
in exploring this option, contact Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).


# Delete or export personal data stored by Unstructured
Source: https://docs.unstructured.io/support/how-to/data-privacy



You can request that Unstructured delete the personal data that Unstructured stores in its systems about you, export that
personal data and send a copy of it to you, or both.

To make a personal data deletion or export request, fill out and submit the
[Data Privacy Request](https://na1.hs-data-privacy.com/request/UYsLQfVEBgaaIvrZb74_EA) webform.

After Unstructured receives your request, within one business day, they will delete the personal data that Unstructured stores in its
systems about you or export that personal data. Unstructured will then send an email to the address listed on the webform—this email will
confirm the deletion or provide a copy of the exported
personal data, respectively. If you have not received this email within 3 business days, you can email Unstructured Support at
[support@unstructured.io](mailto:support@unstructured.io) to check on the status of your request.

If you have any questions about this process or need help filling out or submitting the webform, email Unstructured Support at
[support@unstructured.io](mailto:support@unstructured.io).


# Raise an invoice issue or request a refund
Source: https://docs.unstructured.io/support/how-to/invoice-issue



To raise an issue about an unpaid invoice that Unstructured has sent you, or to request a refund for an invoice that you have already paid, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).

In your email, include the following information:

* The email address of the primary contact for your Unstructured account.
* The related invoice number.
* The monetary amount of the unpaid invoice that you are disputing, or the monetary amount of the already-paid invoice for which you are requesting a refund.
* The specific reasons why you are disputing the charges or requesting a refund.
* Any supporting details, documentation, or other information that helps support your claim.

After Unstructured Support receives your request, within one business day, they will respond with an email to the primary contact, acknowledging your
dispute or refund request. If you have not received this acknowledgement email within 3 business days, you can email Unstructured Support at
[support@unstructured.io](mailto:support@unstructured.io) to check on its status.

The timing and details for resolving the dispute or refund request will depend on the nature of the issue. If you have any follow-up questions about the issue
or just want to check on its current status, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).


# Overview
Source: https://docs.unstructured.io/support/how-to/overview



This section contains information about how to initiate various kinds of self-service support requests for your Unstructured account.

These requests include actions such as requesting Unstructured to delete or export the personal data that Unstructured
has stored about you.

This section does not contain instructions about how to make general support requests. To learn how to make a general support request,
see [Requesting support](/support/request).


# Google Drive source connectors cannot be created by using the API
Source: https://docs.unstructured.io/support/issues/api-connector-secrets



## Issue

When you try to use the [Unstructured API's workflow operations](/api-reference/workflow/overview) to
[create a source connector](/api-reference/api/source/create-source)
that connects [Google Drive](/api-reference/workflow/sources/google-drive) to Unstructured,
an error similar to the following occurs, and the source connector is not created:

```text theme={null}
API error occurred: Status 412. Body: {
    "errors": [
        {
            "field": "access_config.service_account_key",
            "message": "Field is sensitive and must be wrapped in as a secret reference or new secret value"
        }
    ]
}
```

## Cause

This error occurs because the contents of the caller's Google Drive service account's `credentials.json` file (or its
equivalent JSON string representation) must be specially encrypted and formatted before these contents are sent to Unstructured, due to the contents' size.
However, the Unstructured API's workflow operations do not support this special encryption and formatting process.

## Solution

Use the [Unstructured Pipelines](/pipelines/overview) to [create a Google Drive source connector](/pipelines/sources/google-drive) instead, as
Unstructured Pipelines does support this special encryption and formatting process.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Authorization and permissions issues
Source: https://docs.unstructured.io/support/issues/authorization-permissions



## Issues

When you try to connect Unstructured to a specific source or destination, you get one of the following error types:

* `PermissionError`: For example, for Amazon S3, an `Access Denied` error.
* `ClientAuthenticationError`: For example, for Azure Blob Storage, an `AuthenticationFailed` or `Signature not valid` error.
* `AuthError`: For example, for Dropbox, an `expired_access_token` error.
* `ApiPermissionError`: For example, for Confluence, a `permission to view content` error.
* `ClientRequestException`: For example, for OneDrive, an `AccessDenied` error.
* `ValueError`: For example, for Outlook, an `invalid_grant` or `Conditional Access policies` error. For Google Drive, a `File not found related to auth` error.
* `UserError`: For example, for Box, an `Access denied - insufficient permission` error.
* `HttpResponseError`: For example, for Azure Blob Storage, an `AccountIsDisabled` error.

## Possible causes

Unstructured could not access the specified source or destination system due to invalid credentials, insufficient permissions, or restrictive policies. This error can still occur even though the related connector passed Unstructured's connection test.

Possible causes include:

* An incorrect API key, token, password, service account credential, tenant ID, or client ID was specified.
* The specified credentials or access token has expired.
* Insufficient permissions were granted; for example, read or list access is needed for indexing data, or write access is required for uploading data.
* The source's or destination's associated account is disabled or inactive.
* For Microsoft Entra ID, Conditional Access Policies are blocking the authentication flow.
* The specified credentials might be valid, but an incorrect configuration points to a resource that is not authorized for those credentials.

## Possible solutions

* **Verify the credentials**: Double-check all authentication details—keys, secrets, tokens, usernames, passwords, and IDs—for typos and accuracy.
* **Check the expiration**: Ensure that the specified token or key has not expired. Regenerate them if needed.
* **Review permissions**: Confirm that the credentials have the required permissions for the operation—for example, `s3:ListBucket` and `s3:GetObject` for Amazon S3 indexing; `Files.Read.All` and `Sites.Read.All` for OneDrive or SharePoint; write permissions for upload destinations—and grant the necessary roles and permissions in the source and destination systems.
* **Check the status of the account and resource**: Ensure that the user account or service principal is active, and the target resource—such as the Azure Storage Account—is enabled.
* **Check the Conditional Access Policies (for Azure Blob Storage)**: Review the Microsoft Entra ID Conditional Access Policies. They might be blocking non-interactive sign-ins or require specific compliance. Adjust the policies or exclude the Unstructured service principal if appropriate and secure.
* **Reconfigure the connector**: Delete and recreate the source or destination connector configuration in Unstructured with verified credentials.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# S3 destination connector - Unable to locate credentials
Source: https://docs.unstructured.io/support/issues/cannot-locate-credentials



## Issue

When you try to run a workflow in Unstructured, and the workflow contains an Amazon S3 destination connector,
you get an error that contains text similar to one of the following, and the workflow fails to run:

```text theme={null}
500: Error in uploader - [NoCredentialsError] Unable to locate credentials 
```

```text theme={null}
failed to invoke plugin: Unable to locate credentials
```

## Possible causes

* Unstructured tried to save its processed data to the specific Amazon S3 bucket, but Unstructured could not find the necessary
  security keys or other credentials to get access to that bucket.
* The specified AWS Access Key ID and Secret Access Key are missing from your S3 destination connector configuration.
* The specified credentials are incorrect, have been revoked, or have expired.
* The S3 destination connector is configured to use a temporary IAM role, but that role does not have permission to write to the target bucket.

## Possible solutions

* Double-check that your workflow's Amazon S3 destination connector has the correct, valid credentials specified.
* Confirm that the specified credentials are still present and valid in your AWS account.
* If your organization uses IAM roles, contact your platform administrator to ensure the role has `s3:PutObject` permissions for the bucket you are trying to use.

## Additional resources

* Always copy and paste security keys and other credentials carefully.
* If possible, use your organization's secrets manager to handle security keys and other credentials automatically.
* To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Configuration and resource issues
Source: https://docs.unstructured.io/support/issues/configuration-resource



## Issues

When you try to connect Unstructured to a specific source or destination, you get one of the following error types:

* `FileNotFoundError`: For example, for Amazon S3, a `path not found` error.
* `ClientRequestException`: For example, for OneDrive, an `itemNotFound` error.
* `ValueError`: For example, for Google Drive, a `File not found` error. For Amazon S3, an `Invalid endpoint` error.
* `UserError`: For example, for Azure Blob Storage, a `DeploymentNotFound` error.
* `ParamValidationError`: For example, for Amazon S3, an `Invalid bucket name` error.
* `EndpointResolutionError`: For example, for Amazon S3, a `Custom endpoint not valid URI` error.
* `ProgrammingError`: For example, for Snowflake, a `No active warehouse selected` error.
* `UnboundLocalError`: For example, for SharePoint, a `cannot access 'site_drive_item'` error.
* `HTTPError`: For example, for Confluence, a `404 Not Found` error for a specific page or attachment URL.
* `KeyError`: For example, for Jira, an error containing the word 'total'. For Amazon S3, an error containing the word `Key`.

## Possible causes

* Unstructured is configured to interact with a resource—such as a file, path, deployment, endpoint, or database object—that doesn't exist,
  is misnamed, or the configuration itself is invalid.
* There is a typo in a bucket name, folder path, file ID, deployment name, hostname, site path, or database name.
* The specified resource has been deleted or moved.
* An endpoint URL is incorrectly formatted, for example, is missing `https://` or contains invalid characters.
* An Amazon S3 bucket name is not formatted correctly.
* A required configuration is missing in the source or destination connector, for example, there is no active Snowflake warehouse specified.
* An Azure OpenAI deployment name is mismatched, failed, or does not exist.
* An invalid URL is specified for an attachment or a link within a source document.
* There is a specific configuration issue with a connector, for example, the specified SharePoint path does not lead to a valid drive.

## Possible solutions

* **Verify names and paths**: Carefully check all configured names, IDs, and paths—such as for buckets, folders, files, sites, deployments, and endpoint URLs—for accuracy. Ensure they exist in the source and destination system. Case sensitivity often matters.
* **Check formatting**: Ensure that URLs, bucket names, and other parameters adhere to the required format.
* **Verify that the resource exists**: Confirm that the target file, folder, deployment, or other resource exists and has not been moved or deleted.
* **Check configuration dependencies**: Ensure the necessary configurations are set in the source or destination, for example, select and start a Snowflake warehouse by running the `USE WAREHOUSE` command first.
* For **Azure OpenAI**: Double-check that the Deployment Name matches a successful deployment in your Azure portal.
* For **Confluence**: If a `404` error occurs during download, check if the page or attachment link is valid within Confluence itself.
* For **SharePoint**: Verify the Site Path leads to a valid location containing document libraries.
* **Reconfigure the connector**: Review and, as needed, correct any misconfigured settings in the source or destination connector.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Data format, schema, and validation issues
Source: https://docs.unstructured.io/support/issues/data-format-schema-validation



## Issues

When Unstructured tries to process or transfer data, you get one of the following error types:

* `ValidationError`: For example, `S3ConnectionConfig`, `PluginResponse`, or `filedata_meta input type` errors.
* `ClientResponseError`: For example, an `HTTP 422 Unprocessable Entity` error.
* `ListConversionException`: For example, for Pinecone, an `Expected list, got None` error.
* `ServerOperationError`: For example, for Databricks, an `UNRESOLVED_COLUMN` error.
* `WeaviateDeleteManyError`: For example, a `no such prop with name 'record_id'` error.
* `CollectionInsertManyException`: For example, for Astra DB, a `vector dimension mismatch` error.
* `TypeError`: For example, for Pinecone, a `PineconeUploader missing argument` error.
* `UserError`: For example, a wrappingschema validation failure.

## Possible causes

* The data being processed or transferred doesn't match the expected structure, format, or validation rules.
* The data provided in configuration or during processing is not valid.
* The data being sent to a destination doesn't match the destination's schema—for example, the data has missing fields, wrong data types, or an incorrect number of vector dimensions.
* Incorrect configuration values were provided—for example, a non-string token was provided where a string was expected instead.
* There is a schema mismatch between the data generated by Unstructured and the destination schema—for example, missing columns or properties, or wrong data types.
* The specified embedding model generates vectors of a different dimension than the destination index or collection is configured for.
* The data was generated in an unexpected format.

## Possible solutions

* **Verify the data configuration**: Double-check configuration parameters against the relevant documentation for the correct data format and type.
* **Verify the destination schema**: Ensure that the schema—including columns, properties, types, and vector dimensions—in the destination's system matches the data being sent. You might need to update or recreate the destination schema.
* **Check the data fields**: Check if fields such as `record_id`, `element_id`, `text`, `embeddings`, and `metadata` are expected and present.
* **Verify the embedding dimensions**: Confirm that the specified embedding model produces vectors of the dimension expected by the destination collection or index.
* **Contact Unstructured Support**: For `HTTP 422 Unprocessable Entity` errors where the cause isn't clear, `ListConversionException` errors, `TypeError` errors, or persistent schema mismatch issues after verification, [contact Unstructured Support](/support/request).

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Document processing issues
Source: https://docs.unstructured.io/support/issues/document-processing



## Issues

When Unstructured tries to partition or chunk a document, you get one of the following error types:

* `TooManyPageFailuresException`
* `ControllerException`: For example, an error wrapping partitioning or chunking errors such as code `512`.

## Possible causes

* There are issues with the document's underlying content or structure.
* Unstructured encountered too many errors while processing individual pages or elements within a specific document, exceeding some internal failure threshold.
* The document is corrupted or malformed. This is especially the case for some complex PDFs.
* The document has a highly unusual underlying structure or has highly unusual content, which the specified partitioning model cannot handle.
* The document is encrypted or password-protected.
* Some underlying issues are causing page-level failures. This could sometimes be related to quotas if a per-page VLM is used.

## Possible solutions

* **Inspect the document**: Examine the specific document. Make sure it opens correctly, does not seem to look unusual, and does not appear to be corrupted or encrypted.
* **Test a simpler document**: Try processing a known-good, simple document of the same type, to see if the error is document-specific.
* **Check quotas (if VLM related)**: If the underlying errors mention quotas, see [Quota, billing, and rate limiting issues](/support/issues/quota-billing-rate-limiting).
* **Report the problematic document to Unstructured Support**: If the issue seems specific to an otherwise seemingly valid document, [report it to Unstructured Support](/support/request). If possible, provide the document for Unstructured to further investigate.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Unable to sign up with personal email address
Source: https://docs.unstructured.io/support/issues/email-addresses



## Issue

When you try to [sign up for a free Unstructured account](https://unstructured.io/?modal=try-for-free), and you enter a personal email address into
the **Company email** field in the sign-up form, the following error message appears:

```text theme={null}
Please enter a different email address. This form does not accept addresses from <domain>.
```

This error message prevents you from clicking **Submit** to sign up for a free Unstructured account.

## Cause

Unstructured does not accept email addresses from free domains (such as Gmail, Hotmail, Outlook, and Yahoo) when signing up for a free Unstructured account.

## Solution

Use a company email address when signing up for a free Unstructured account.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Get authenticated user error
Source: https://docs.unstructured.io/support/issues/get-authenticated-user-error



## Issue

When you try to sign in to Unstructured, you get an error that contains text similar to the following:

```text theme={null}
Error in getAuthenticatedUser
```

This error can prevent you from accessing your
Unstructured account dashboard or performing other actions in your Unstructured account.

## Possible causes

* Unstructured is having trouble verifying your login session.
* Your Unstructured login session has expired.
* Your web browser's cache or cookies are causing some kind of conflict.
* There is a temporary issue with Unstructured's authentication service.

## Possible solutions

* Log out of your Unstructured account completely, if you can. To do this, in Unstructured Pipelines, on the sidebar,
  click your user icon, and then click **Logout**.
* Clear your web browser's cache and cookies for the Unstructured website.
* Close and then reopen your web browser, and then try to sign in again to your Unstructured account.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Google Drive schema validation failure
Source: https://docs.unstructured.io/support/issues/google-drive-schema-validation



## Issue

When you save and test a Google Drive source connector, you get an error that contains text similar to the following:

```text theme={null}
Schema validation failure for GoogleDriveConnectionConfig: 
Value error, Expecting value: line 1 column 1 (char 0) at 
access_config,service_account_key got -----BEGIN PRIVATE KEY-----
...
```

Although you can finish creating and saving this connector, the connection test fails, and you cannot successfully run any
workflows that use this failed connector.

## Cause

You have configured the Google Drive source connector with the Google Drive service account's key in an incorrect format.
You likely have specified only a portion of the file, such as just the private key part. Unstructured expects the entire contents
of the Google Drive service account's key file.

## Solution

1. Find the original Google Drive service account's key file in JSON format, which you downloaded earlier from your Google Cloud account when
   you originally created the Google Drive source connector.
2. Open the file in a text editor such as Notepad, Visual Studio Code, or TextEdit.
3. Copy everything in the file, starting from the first opening brace `{` to the final closing brace `}`.
4. In the original Google Drive source connector's settings, delete the original contents of the **Service Account Key** field, and paste
   the complete JSON content that you just copied.
5. Save the changes and re-test and re-save the connector.

## Additional resources

* If the connector test still fails, try using a tool such as `tr` for macOS or Linux, or PowerShell for Windows, to copy
  the entire contents of the file, instead of a text editor, as follows:

  * For macOS or Linux:

    ```text theme={null}
    tr -d '\n' < <path-to-downloaded-key-file>
    ```

  * For Windows:

    ```text theme={null}
    (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
    ```

* To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Internal issues
Source: https://docs.unstructured.io/support/issues/internal-file-handling



## Issues

Unstructured returns one of the following error types:

* `NotFoundException`: For example, an error retrieving intermediate file data from internal storage, with a pattern such as `/f8c6.../element_dicts/...json`.
* `FileNotFoundError`: For example, an error with internal system paths that use a pattern such as `/home/etl/node/staged/...` or `/home/etl/node/downloads/...`.
* `ValueError`: For example, a `Failed to decrypt secret` error.
* `AttributeError`: For example, a `'TimeoutError' object has no attribute 'status_code'` error.
* `asyncio.exceptions.CancelledError`

## Possible causes

* There is an error in a previous workflow or ingestion pipeline stage within Unstructured's environment that is preventing intermediate files from being created correctly.
* There are file system or storage issues within Unstructured's environment.
* There are problems with Unstructured's internal management of secret keys or configurations.
* There are issues with Unstructured's internal error handling or service lifecycle management.
* There are some other general issues with Unstructured's internal workings, often related to managing temporary files or internal state.

## Possible solutions

* **Retry the job**: Retry the entire job, as the issue might have been temporary.
* **Contact Unstructured Support**: These issues are generally fixable only by Unstructured. These issues typically indicate internal issues that customers cannot fix themselves.
  Be sure when you [contact Unstructured Support](/support/request) to include the full error message, timestamp, and job details.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Network, connection, and timeout issues
Source: https://docs.unstructured.io/support/issues/network-connection-timeout



## Issues

When you try to establish or maintain a network connection between Unstructured and an external system, you get one of the following error types:

* `ReadTimeout`
* `TimeoutError`
* `ServerDisconnectedError`
* `ClientConnectorError`: For example, a `cannot connect to localhost` error.
* `ConnectionError`: For example, a `Failed to resolve 'model-registry-api...` error.
* `ServiceResponseError`: For example, for Azure, a `Timeout on reading data from socket` error.
* `ClientPayloadError`: For example, a `Response payload not completed` or `Connection reset by peer` error.
* `OSError`: For example, for Amazon S3 during a file upload, an `[Errno 22] The request body terminated unexpectedly` error.

## Possible causes

* Unstructured failed to connect to a required system.
* The connection timed out while waiting for a response.
* An established connection was unexpectedly closed.
* Transient network issues were encountered with the local network, with the Internet, or with a cloud provider.
* Firewalls are blocking connections.
* Some services are temporarily unavailable, slow, or unresponsive.
* The processing of very large files is causing long-running operations, which lead to exceeding default timeouts.
* Issues with DNS resolutions were encountered.

## Possible solutions

* **Retry the job**: Many network errors are transient. Wait a few minutes and then retry the job.
* **Check network connectivity**: Ensure stable network connectivity with the related source and destination systems.
* **Check firewalls**: Verify that any firewalls are not blocking necessary connections to source or destination APIs, blob storage, databases, vector stores, or ports.
* **Process smaller batches or files**: If timeouts occur during processing or uploading very large files, try using smaller files or batches.
* **Contact Unstructured Support**: For persistent `ClientConnectorError`, `ConnectionError`, or other frequent timeouts or disconnects that aren't resolved by retries, [contact Unstructured Support](/support/request). These issues can sometimes point to internal Unstructured system problems.

## Additional resources

* [Unstructured Status](https://status.unstructured.io) dashboard
* [Request Unstructured support](/support/request)


# The Fast strategy is not available for image files
Source: https://docs.unstructured.io/support/issues/no-fast-partitioning-for-images



## Issue

When you try to run a workflow in Unstructured with the following settings:

* The workflow contains a **Partitioner** node set to use the **Fast** partitioning strategy.
* The workflow is configured to process an image file, such as a PNG or JPG file.

You get an error that contains text similar to the following, and the workflow fails to run:

```text theme={null}
422: Error in partition - {"detail":"The fast strategy is not available for image files"}
```

## Cause

Unstructured is trying to process an image file—such as a PNG or JPG file—by using the **Fast** partitioning strategy, but
this strategy is designed for text-based documents only and cannot analyze the layout of images.

## Solution

1. Change the workflow's **Partitioner** node's **Partition Strategy** setting to **Auto**, **High Res**, or **VLM**.
2. Re-run the workflow.

## Additional resources

* If your workflow is configured to process a mixture of file types, you should use the **Auto** partitioning strategy. Unstructured will then
  automatically choose the best partitioning strategy for each file.
* Learn more about the various [partitioning](/concepts/partitioning) strategies.
* To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Overview
Source: https://docs.unstructured.io/support/issues/overview



This section contains solutions to common issues that you might encounter as you use Unstructured.

Each issue contains the following sections:

* **Issue**: A description of the issue. Use this section to confirm that you are experiencing the same issue that is being described.
* **Cause**: The cause of the issue. Use this section to help you understand why the issue is (or might be) occurring. In some cases,
  there might be multiple causes, or the causes might be indirect or dependent on other factors. In these cases, the title will be
  labeled as **Possible causes** instead.
* **Solution**: A solution to the issue. In some cases, there might be solutions that are dependent on other factors. In these cases,
  the title will be labeled as **Possible solutions** instead.
* **Additional resources**: Any usage notes, links, or other secondary information that might also help you to resolve the issue.

To ask questions or get additional help with these issues, see [requesting support](/support/request).


# Quota, billing, and rate limiting issues
Source: https://docs.unstructured.io/support/issues/quota-billing-rate-limiting



## Issues

When you try to use Unstructured, you get one of the following error types:

* `ClientResponseError`: For example, an `HTTP 402 Payment Required` error.
* `SDKError`: For example, an `Insufficient API quota` error.
* `PineconeApiException`: For example, an `HTTP 429 Too Many Requests` or `write unit limit reached` error.
* `DestinationConnectionError`: For example, an error that wraps an `HTTP 429 Too Many Requests` error.
* `HTTPError`: For example, an `HTTP 429 Too Many Requests` error.
* `Error`: For example, a `RESOURCE_EXHAUSTED` or a `reduce request rate` error.

## Possible causes

* Unstructured or the source's or destination's system imposes usage limits such as the number of API calls or data processed or stored over a specified time period or a specified rate of requests. These limits have been reached, or payment is required to continue.
* A daily or monthly API quota for Unstructured, the source's or destination's API, or a specified model, has been reached.
* An API rate limit of requests per second or minute has been reached on the source's or destination's system.
* A subscription issue or required payment blocker has been encountered by Unstructured or for a related service.

## Possible solutions

Some Unstructured **Business** account plans for dedicated instances or in-VPC deployments require the customer to set up or pay for related
services, features, and models outside of Unstructured, in order to achieve volume discounts and to better manage overall usage costs.
If you are on one of these account plans, check the following:

* **Check your usage and quotas**: Review your current usage against your budget limits in your Unstructured account dashboard and any relevant third-party services—such as Pinecone, Azure OpenAI, or Databricks.
* **Verify your billing and subscription details**: Ensure any billing agreements and subscriptions are active and their payments are up-to-date.

Other possible solutions include the following:

* **Reduce the rate of requests if you can**.
* **Run processing jobs less frequently if possible**.
* **Process data in smaller batches if you can**.
* **If configurable, reduce the concurrency settings in your Unstructured ingestion pipelines**.
* **Try processing jobs again later**. Some rate limits are often time-based; waiting and retrying later might work.
* **Upgrade your plan**. If you are consistently hitting limits, consider upgrading your plan for Unstructured or the third-party service.
* **Contact Unstructured Support**. If you are consistently hitting limits, consider [contacting Unstructured Support](/support/request) to discuss your options.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Workflow already has a job scheduled or in progress
Source: https://docs.unstructured.io/support/issues/workflow-job-in-progress



## Issue

When you try to run a workflow, you get a warning message that contains text similar to the following:

```text theme={null}
Failed to schedule job for workflow ... Workflow already has a job scheduled or in progress. Retrying in 60 seconds.
```

## Causes

* Unstructured tried to start a new job run for the workflow, but the previous job run for that workflow is stuck or has otherwise not finished yet.
* The workflow's schedule is too frequent for how long the job takes to run. For example, the workflow might be scheduled to
  run every 5 minutes, but each job run for that workflow typically takes 10 minutes or longer to complete.

## Solutions

* Unstructured will wait and try the new job run again in 60 seconds.
* Check previous job runs for the workflow to see if any of them are stuck. If so, try to cancel them.
* Check previous job runs for the workflow to see if any of them seem to be running much longer than usual. If so,
  consider adjusting the workflow's schedule to run less frequently. Or, temporarily turn off the workflow's schedule and then run the workflow manually as needed.

## Additional resources

To ask questions or get additional help with this issue, see [requesting support](/support/request).


# Requesting support
Source: https://docs.unstructured.io/support/request



<Tip>
  If you are having an issue with Unstructured, try checking the [Issues](/support/issues/overview) section first
  to see if your issue has already been addressed.
</Tip>

To request support, do the following:

* If you are using the [Unstructured open source library](/open-source/introduction/overview),
  request support by creating a new message in the **#general**
  channel in the [Unstructured Slack community](https://short.unstructured.io/pzw05l7).
* For Unstructured software-as-a-service (Saas) accounts, including legacy pay-as-you-go,
  [Let's Go](https://unstructured.io/pricing) and [Pay-As-You-Go](https://unstructured.io/pricing) accounts,
  email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).<sup>\*</sup>
* For Unstructured [Business](/business/overview) accounts,
  contact your assigned Unstructured Account Executive (AE) or Customer Success Manager (CSM) for access to your
  dedicated Customer Support Portal.
  If you do not know who your assigned AE or CSM is,
  email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).<sup>\*</sup>

It is critical that you route your support requests appropriately, as follows. Incorrect routing can result in delayed or no response, and
any related service level agreement (SLA) commitments by Unstructured will not be honored.

* **Support requests for paid Unstructured accounts (except for **Business** accounts) must only be emailed to Unstructured Support at
  [support@unstructured.io](mailto:support@unstructured.io)**. These kinds of support requests, if submitted by any other means,
  might take longer to receive a response from Unstructured Support—or they might not be responded to at all.
* **Support requests for Unstructured **Business** accounts must only be directed to your
  assigned Unstructured Account Executive (AE) or Customer Success Manager (CSM).** These kinds of support requests, if submitted by any other means,
  might take longer to receive a response from Unstructured Support—or they might not be responded to at all.
* **Support requests for the Unstructured open source library must only be posted in the #general channel in the Unstructured Slack community**.
  These kinds of support requests, if submitted by any other means, will not be responded to.

<sup>\*</sup> Before contacting Unstructured Support, please ensure that you consult the
latest version of the [Unstructured Shared Responsibility Model](/support/shared-responsibility) to check
who owns resolution of the issue you are experiencing. If you contact Unstructured Support about an issue for
which the Model states that the customer is ultimately responsible, Support could return the unresolved issue
back to you and take no further action.


# Shared Responsibility Model
Source: https://docs.unstructured.io/support/shared-responsibility



The Unstructured Shared Responsibility Model defines the division of ownership and duties between
Unstructured, customers, and the cloud service providers (CSPs) that Unstructured and customers rely on.

The Model lists and clarifies who is ultimately responsible for specific aspects of
physical and cloud infrastructure, security, and application services, ensuring each party understands
not only its own obligations but the obligations of the other participating parties. This Model is
regularly updated to reflect the latest additions and changes to Unstructured's and the
CSPs' product and service offerings.

Before you contact Unstructured Support, you are encouraged to review the latest version of the Model
to check who owns resolution of the issue you are experiencing. If you contact
Unstructured Support about an issue for which the Model states that the customer is ultimately responsible,
Support could return the unresolved issue back to you and take no further action.

To see the latest version of the Model, go to the [Unstructured Trust Portal](https://trust.unstructured.io/),
scroll down to the "Topics and common questions" section, and expand the entry labelled
"Shared Responsibility Model."


# Check service status
Source: https://docs.unstructured.io/support/status



The [Unstructured Status](https://unstructuredio.trust.pagerduty.com/posts/dashboard) dashboard provides information about the
status of the Unstructured service.

<Note>
  This dashboard provides information only about the publicly available Unstructured service. If you are running a
  self-contained distribution of Unstructured, contact your organization's network administrator instead.
</Note>

From this dashboard, you can:

* Check the status of the publicly available Unstructured Pipelines. This is labeled in the dashboard as **Platform Frontend**.
* Check the status of the publicly available Unstructured API. This is labeled in the dashboard as **Platform API**.
* Subscribe to public service status updates delivered to an email address for your organization, to a specific channel in your organization's Slack workspace, or
  to a notification URL provided by your organization. To do this, click the **Subscribe to Updates** button and then follow the on-screen prompts.

You can also [browse the history](https://unstructuredio.trust.pagerduty.com/posts/history) of previous public service status updates.


# Welcome to Unstructured!
Source: https://docs.unstructured.io/welcome



Unstructured turns your organization's unstructured documents and semi-structured data into structured, AI-ready data at scale with high performance, accuracy, and relevance.

<img alt="Unstructured conceptual data flow" />

[Learn about Unstructured's product offerings](/about).

<Tip>
  Are you using LLMs or AI agents to read this documentation site's contents? For better results,
  connect them to this site's
  [llms.txt](https://docs.unstructured.io/llms.txt) and
  [llms-full.txt](https://docs.unstructured.io/llms-full.txt) files.
  [Learn more about llms.txt and llms-full.txt files](https://www.mintlify.com/blog/the-value-of-llms-txt-hype-or-real).
</Tip>

## <Icon icon="person-running" />    Getting started

Go to one of the following, based on your user profile:

|                                         |                                                                   |
| --------------------------------------- | ----------------------------------------------------------------- |
| An individual developer                 | [Unstructured API quickstart](/api-reference/quickstart/overview) |
| An individual user                      | [Unstructured Pipelines quickstarts](/pipelines/quickstart)       |
| A team, business, or other organization | [Unstructured Business overview](/business/overview)              |

## <Icon icon="question" />    Questions? Need help?

For general questions about Unstructured's product offerings, email Unstructured Sales at [sales@unstructured.io](mailto:sales@unstructured.io).

For Unstructured technical support, see [Requesting support](/support/request).