Unstructured

1. Sign in to your Unstructured account, at https://app.unstructured.io.
2. Click Google or GitHub to sign in with your Google or GitHub account. Or, enter your email address and then click Sign In.
3. If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the Sign In link.
4. The first time you sign in, read the terms and conditions, and then click Accept.

• From your Unstructured account dashboard, in the sidebar, click API > Platform.

1. From your Unstructured Platform dashboard, in the sidebar, click Connectors.
2. Click Sources.
3. Click Add new.
4. For Name, enter some unique name for this connector.
5. In the Provider area, click the source location type that matches yours.
6. Click Continue.
7. Fill in the fields with the appropriate settings. Learn more.
8. If a Continue button appears, click it, and fill in any additional settings fields.
9. Click Save and Test.

1. In the sidebar, click Connectors.
2. Click Destinations.
3. Click Add new.
4. For Name, enter some unique name for this connector.
5. In the Provider area, click the destination location type that matches yours.
6. Click Continue.
7. Fill in the fields with the appropriate settings. Learn more.
8. If a Continue button appears, click it, and fill in any additional settings fields.
9. Click Save and Test.

1. In the sidebar, click Workflows.

2. Click New Workflow.

3. Next to Build it with me, click Create Workflow.

   If a radio button appears instead of Build it with me, select it, and then click Continue.

4. For Workflow Name, enter some unique name for this workflow.

5. In the Sources dropdown list, select your source location from Step 3.

6. In the Destinations dropdown list, select your destination location from Step 4.

   You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.

7. Click Continue.

8. In the Optimize for section, select the option to choose one of these predefined workflow settings groups:

   • Basic is a good choice if you have text-only documents that have no images or tables in them.
   • Advanced is a good choice if you have complex documents that have images or tables or both in them.

9. The Reprocess all box applies only to the Amazon S3 and Azure Blob Storage source connectors:

   • Checking this box reprocesses all documents in the source location on every workflow run.
   • Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents’ contents change.

10. If you want to retry processing any documents that failed to process, check the Retry Failed Documents box.

11. Click Continue.

12. If you want this workflow to run on a schedule, in the Repeat Run dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select Don’t repeat.

13. Click Complete.

1. If you did not choose to run this workflow on a schedule in Step 5, you can run the workflow now: on the sidebar, click Workflows.
2. Next to your workflow from Step 5, click Run.

1. In the sidebar, click Jobs.
2. In the list of jobs, wait for the job’s Status to change to Finished.
3. Click the row for the job.
4. If Overview displays Success, go to the next Step.

Go to your destination location to view the processed data.

Sign up for a 14-day trial of the Unstructured Serverless API at https://app.unstructured.io.

1. Enter your email address and then click Sign In to receive a magic link to sign in to your personalized dashboard, or authenticate yourself with your Google or GitHub account.
2. Once you authenticate with the magic link or with your Google or GitHub account, your dashboard appears.

1. On the sidebar, click API Keys, if it is not already selected.
2. To get your API key, click the copy icon in the Actions column for your API key, and then click Key Only. Store your copied API key in a secure location. Do not share it with others.
3. To get your API URL, click the copy icon next to the URL next to API URL. Store your copied API URL in a secure location. Do not share it with others.

1. Set an environment variable named UNSTRUCTURED_API_KEY to the value of your Unstructured API key.
2. Set another environment variable named UNSTRUCTURED_API_URL to the value of your Unstructured API URL.

Run the following command:

pip install unstructured-ingest

You might also need to install additional dependencies, depending on your needs. Learn more.

Run the following code, replacing:

• <path/to/input> with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
• <path/to/output> with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

import os

from unstructured_ingest.v2.pipeline.pipeline import Pipeline
from unstructured_ingest.v2.interfaces import ProcessorConfig
from unstructured_ingest.v2.processes.connectors.local import (
    LocalIndexerConfig,
    LocalDownloaderConfig,
    LocalConnectionConfig,
    LocalUploaderConfig
)
from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

if __name__ == "__main__":
    Pipeline.from_configs(
        context=ProcessorConfig(),
        indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
        downloader_config=LocalDownloaderConfig(),
        source_connection_config=LocalConnectionConfig(),
        partitioner_config=PartitionerConfig(
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
            strategy="hi_res",
            additional_partition_args={
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        ),
        uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
    ).run()

Go to your destination location to view the processed data.