The Unstructured Ingest CLI enables you to use command-line scripts to send files in batches to Unstructured for processing, and to tell Unstructured where to deliver the processed data. Learn more.

The Unstructured Ingest CLI does not work with the Unstructured API.

For information about the Unstructured API, see the Unstructured API Overview.

​
Getting started

You can use the Unstructured Ingest CLI to process files locally, or you can use the Ingest CLI to send files in batches to Unstructured for processing.

Local processing does not use an Unstructured API key or API URL.

Using the Ingest CLI to send files in batches to Unstructured for processing is more robust, and usage is billed to you on a pay-as-you-go basis. Usage requires an Unstructured API key and API URL, as follows:

  1. Sign in to your Unstructured account:

    • If you do not already have an Unstructured account, go to https://unstructured.io/contact and fill out the online form to indicate your interest.
    • If you already have an Unstructured account, sign in by using the URL of the sign in page that Unstructured provided to you when your Unstructured account was created. After you sign in, the Unstructured user interface (UI) then appears, and you can start using it right away. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.

  2. Get your Unstructured API key:

    a. In the Unstructured UI, click API Keys on the sidebar.
    b. Click Generate API Key.
    c. Follow the on-screen instructions to finish generating the key.
    d. Click the Copy icon next to your new key to add the key to your system’s clipboard. If you lose this key, simply return and click the Copy icon again.

  1. The Unstructured API URL for Unstructured Ingest was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.

The default URL for Unstructured Ingest is the same as the default URL for the Unstructured Partition Endpoint: https://api.unstructuredapp.io/general/v0/general. However, you should always use the URL that was provided to you when your Unstructured account was created.

​
Installation

One approach to get started quickly with the Unstructured Ingest CLI is to install Python and then run the following command:

pip install unstructured-ingest

This default installation option enables the ingestion of plain text files, HTML, XML, JSON and emails that do not require any extra dependencies. This default option also enables you to specify local source and destination locations.

You might also need to install additional dependencies, depending on your needs. Learn more.

For additional installation options, see Unstructured Ingest CLI in the Ingest section.

To migrate from older, deprecated versions of the Ingest CLI that used pip install unstructured, see the migration guide.

​
Usage

To call the Unstructured Ingest CLI, follow this calling pattern, where:

  • <source> is the command name for one of the available source (input) connectors, such as local for a local source location, azure for an Azure Storage account source, s3 for an Amazon S3 bucket source, and so on.
  • <destination> is the command name for one of the available destination (output) connectors, such as local for a local destination, azure for an Azure Storage account destination, s3 for an Amazon S3 bucket destination, and so on.
  • <setting> is one or more command-line options for specifying how and where Unstructured will ingest the files from the <source>, or how and where to deliver the processed data to the <destination>.
CLI
#!/usr/bin/env bash

unstructured-ingest \
  <source> \
    --<setting1> <value1> \
    --<setting2> <value2> \
    --<settingN> <valueN> \
  <destination> \
    --<setting1> <value1> \
    --<setting2> <value2> \
    --<settingN> <valueN>

To learn how to use the Unstructured Ingest CLI to work with a specific source (input) and destination (output) location, see the CLI script examples for the source and destination connectors that are available for you to choose from.

See also the ingest configuration settings for command-line options that enable you to further control how batches are sent and processed.