Skip to main content
To start using the Unstructured Pipeline API right away, skip ahead to the quickstart or walkthrough, which uses the Unstructured Python SDK from remotely hosted Google Colab notebooks. These notebooks require no local setup.
The Unstructured Pipelines workflow designer features a no-code user interface for transforming your unstructured data into data that is ready for retrieval-augmented generation (RAG). The Pipeline API enable a full range of partitioning, chunking, embedding, and enrichment options for your files and data. It is designed to batch-process files and data in remote locations; send processed results to various storage, databases, and vector stores; and use the latest and highest-performing models. The Pipeline API has built-in logic to deliver the highest quality results at the lowest cost. This page provides an overview of the Pipeline API. These operations enable pipeline automation usage scenarios as well as for documentation, reporting, and recovery needs.

​
Getting started

Choose one of the following options to get started with the Pipeline API:
  • Follow the quickstart or walkthrough, which use the Unstructured Python SDK from remotely hosted Google Colab notebooks. These notebooks require no local setup.
  • Start using the Unstructred Python SDK. This option requires you to set up a local Python virtual environment.
  • Start using a local REST client, such as curl or Postman.

​
Quickstart

The following quickstart shows how to use the Unstructured Python SDK to have Unstructured process local files by using the Unstructured Pipeline API’s local-file processing jobs functionality. This quickstart is available in two options:
  • Use a remote notebook - This option uses a remotely hosted Google Colab notebook. There are no additional setup steps required.
  • Use your local machine - This option requires you to install the Unstructured Python SDK on your local machine.
The local-file processing jobs functionality is designed to work only by processing local files.To process files (and data) in remote file and blob storage, databases, and vector stores, you must use other operations in the Pipeline API. To learn how, see for example the notebook Dropbox-To-Pinecone Connector API Quickstart for Unstructured.

​
Walkthrough

This walkthrough builds upon the quickstart. Like the quickstart, this walkthrough shows how to use the Unstructured Python SDK to have Unstructured process local files by using the Pipeline API’s local-file processing jobs functionality. This walkthrough goes further by also showing in depth how to use Unstructured’s chunking, enriching, and embedding features. This walkthrough is available as a remotely hosted Google Colab notebook. This notebook requires no local setup.
The local-file processing jobs functionality is designed to work only by processing local files.To process files (and data) in remote file and blob storage, databases, and vector stores, you must use other operations in the Pipeline API. To learn how, see for example the notebook Dropbox-To-Pinecone Connector API Quickstart for Unstructured.

​
Unstructured Python SDK

Watch the following 4-minute video to learn how to use the Python SDK to call the Pipeline API to create connectors.
Watch the following 4-minute video to learn how to use the Python SDK to call the Pipeline API to create workflows and jobs.
Open a related notebook that covers many of the concepts that are shown in the preceding videos. The Unstructured Python SDK, beginning with version 0.30.6, allows you to call the Pipeline API through standard Python code. To install the Unstructured Python SDK, run the following command from within your Python virtual environment: 
pip install "unstructured-client>=0.30.6"
If you already have the Unstructured Python SDK installed, upgrade to at least version 0.30.6 by running the following command instead: 
pip install --upgrade "unstructured-client>=0.30.6"
The Unstructured Python SDK code examples, shown later on this page and on related pages, use the following environment variable, which you can set as follows: 
export UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
This environment variable enables you to more easily run the following Unstructured Python SDK examples and help prevent you from storing scripts that contain sensitive API keys in public source code repositories. To get your Unstructured API key, do the following:
  1. If you do not already have an Unstructured account, sign up for free. After you sign up, you are automatically signed in to your new Unstructured Let’s Go account, at https://platform.unstructured.io.
    To sign up for a Business account instead, contact Unstructured Sales, or learn more.
  2. If you have an Unstructured Let’s Go, Pay-As-You-Go, or Business SaaS account and are not already signed in, sign in to your account at https://platform.unstructured.io.
    For other types of Business accounts, see your Unstructured account administrator for sign-in instructions, or request support.
  3. Get your Unstructured API key:
     a. After you sign in to your Unstructured Let’s Go, Pay-As-You-Go, or Business account, click API Keys on the sidebar.
    For a Business account, before you click API Keys, make sure you have selected the organizational workspace you want to create an API key for. Each API key works with one and only one organizational workspace. Learn more.
    b. Click Generate API Key.
    c. Follow the on-screen instructions to finish generating the key.
    d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
Calls made by the Unstructured Python SDK’s unstructured_client functions for creating, listing, updating, and deleting connectors, workflows, and jobs in Unstructured Pipelines all use the Pipeline API URL. This URL was provided to you when your Unstructured account was created. If you do not have this URL, request support.
The default URL for the Pipeline API is https://platform.unstructuredapp.io/api/v1. However, you should always use the URL that was provided to you when your Unstructured account was created.
To specify an API URL in your code, set the server_url parameter in the UnstructuredClient constructor to the target API URL. The Pipeline API enables you to work with connectors, workflows, and jobs in Unstructured Pipelines.
  • A source connector ingests files or data into Unstructured from a source location.
  • A destination connector sends the processed data from Unstructured to a destination location.
  • A workflow defines how Unstructured will process the data.
  • A job runs a workflow at a specific point in time.
For general information about these objects, see: Skip ahead to start learning about how to use the Unstructured Python SDK to work with connectors, workflows, and jobs programmatically.

​
REST endpoints

The Pipeline API is callable from a set of Representational State Transfer (REST) endpoints, which you can call through standard REST-enabled utilities, tools, programming languages, packages, and libraries. The examples, shown later on this page and on related pages, describe how to call the Pipeline API with curl and Postman. You can adapt this information as needed for your preferred programming languages and libraries, for example by using the requests library with Python.
You can also use the Pipeline API - Swagger UI to call the REST endpoints that are available through the default Pipeline API URL: https://platform.unstructuredapp.io. To use the Swagger UI, you must provide your Unstructured API key with each call. To get this API key, see the quickstart, earlier on this page.Note that you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, request support.

​
curl and Postman

The following curl examples use the following environment variables, which you can set as follows: 
export UNSTRUCTURED_API_URL="https://platform.unstructuredapp.io/api/v1"
export UNSTRUCTURED_API_KEY="<your-unstructured-api-key>"
For the API URL, this URL was provided to you when your Unstructured account was created. If you do not have this URL, request support.
The default URL for the Pipeline API is https://platform.unstructuredapp.io/api/v1. However, you should always use the URL that was provided to you when your Unstructured account was created.
These environment variables enable you to more easily run the following curl examples and help prevent you from storing scripts that contain sensitive URLs and API keys in public source code repositories. To get your Unstructured API key, do the following:
  1. If you do not already have an Unstructured account, sign up for free. After you sign up, you are automatically signed in to your new Unstructured Let’s Go account, at https://platform.unstructured.io.
    To sign up for a Business account instead, contact Unstructured Sales, or learn more.
  2. If you have an Unstructured Let’s Go, Pay-As-You-Go, or Business SaaS account and are not already signed in, sign in to your account at https://platform.unstructured.io.
    For other types of Business accounts, see your Unstructured account administrator for sign-in instructions, or request support.
  3. Get your Unstructured API key:
     a. After you sign in to your Unstructured Let’s Go, Pay-As-You-Go, or Business account, click API Keys on the sidebar.
    For a Business account, before you click API Keys, make sure you have selected the organizational workspace you want to create an API key for. Each API key works with one and only one organizational workspace. Learn more.
    b. Click Generate API Key.
    c. Follow the on-screen instructions to finish generating the key.
    d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
The following Postman examples use variables, which you can set as follows:
  1. In Postman, on your workspace’s sidebar, click Environments.
  2. Click Globals.
  3. Create two global variables with the following settings:
    • Variable: UNSTRUCTURED_API_URL
    • Type: default
    • Initial value: The Pipeline API URL that was provided to you when your Unstructured account was created.
    • Current value: The Pipeline API URL that was provided to you when your Unstructured account was created.

    • Variable: UNSTRUCTURED_API_KEY
    • Type: secret
    • Initial value: <your-unstructured-api-key>
    • Current value: <your-unstructured-api-key>
  4. Click Save.
These variables enable you to more easily run the following examples in Postman and help prevent you from storing Postman collections that contain sensitive URLs and API keys in public source code repositories. Unstructured offers a Postman collection that you can import into Postman to make Pipeline API requests through a graphical user interface.
  1. Install Postman.
  2. Sign in to Postman.
  3. In your workspace, click Import. Import a Postman collection
  4. In the Paste cURL, Raw text or URL box, enter the following URL, and then press Enter: For all Pipeline API requests: 
    https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/postman/Unstructured-REST-API-Workflow-Endpoint.postman_collection.json
    For local-file processing job related operations only: 
    https://raw.githubusercontent.com/Unstructured-IO/docs-samples/main/postman/Unstructured-REST-API-Local-File-Jobs.postman_collection.json
    Each local-file processing job is limited to 10 files, and each file is limited to 50 MB in size.If you need to launch a series of local-file processing jobs in rapid succession, you must wait at least one second between launch requests. Otherwise, you will receive a rate limit error.A maximum of five local-file processing jobs can be running in your Unstructured account. If you launch a new local-file processing job but 5 existing local-file processing jobs are still running, the new local-file processing job will remain in a scheduled state until one of the five existing local-file processing jobs is done running.
  5. On the sidebar, click Collections.
  6. Expand Unstructured REST API - Workflow Endpoints.
  7. Select the request that you want to use.
  8. As applicable, modify the URL as needed to specify any required resource IDs for the request.
  9. On the Headers tab, next to unstructured-api-key, enter your Unstructured API key in the Value column. As applicable, add, remove, or modify any other required headers for the request.
  10. As applicable, on the Params tab, add, remove, or modify any required parameters for the request.
  11. As applicable, on the Body tab, add, remove, or modify the required request body for the request.
  12. Click Send.
  13. To save the response, in the response area, click the ellipses, and then click Save response to file.
To get your Unstructured API key, do the following:
  1. If you do not already have an Unstructured account, sign up for free. After you sign up, you are automatically signed in to your new Unstructured Let’s Go account, at https://platform.unstructured.io.
    To sign up for a Business account instead, contact Unstructured Sales, or learn more.
  2. If you have an Unstructured Let’s Go, Pay-As-You-Go, or Business SaaS account and are not already signed in, sign in to your account at https://platform.unstructured.io.
    For other types of Business accounts, see your Unstructured account administrator for sign-in instructions, or request support.
  3. Get your Unstructured API key:
     a. After you sign in to your Unstructured Let’s Go, Pay-As-You-Go, or Business account, click API Keys on the sidebar.
    For a Business account, before you click API Keys, make sure you have selected the organizational workspace you want to create an API key for. Each API key works with one and only one organizational workspace. Learn more.
    b. Click Generate API Key.
    c. Follow the on-screen instructions to finish generating the key.
    d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.
The Pipeline API enables you to work with connectors, workflows, and jobs in Unstructured Pipelines.
  • A source connector ingests files or data into Unstructured from a source location.
  • A destination connector sends the processed data from Unstructured to a destination location.
  • A workflow defines how Unstructured will process the data.
  • A job runs a workflow at a specific point in time.
For general information about these objects, see: Skip ahead to start learning about how to use the REST endpoints to work with connectors, workflows, and jobs programmatically.

​
Restrictions

The following Unstructured SDKs, tools, and libraries do not work with the Pipeline API: The following Unstructured API URL is also not supported: https://api.unstructuredapp.io/general/v0/general (the default legacy Unstructured Partition Endpoint URL).