Unstructured

To sign up for the Unstructured Platform, go to the For Developers page and enter your information.

1. After you have signed up through the For Developers page, the Unstructured Platform sign-in page appears.

   If you signed up through the For Enterprise page instead, your sign-in process will be different. For enterprise sign-in guidance, contact Unstructured Sales at sales@unstructured.io.

2. Click Google or GitHub to sign in with the Google or GitHub account that you signed up with through the For Developers page. Or, enter the email address that you signed up with, and then click Sign In.

3. If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the Sign In link.

4. The first time you sign in, read the terms and conditions, and then click Accept.

1. From your Unstructured Platform dashboard, in the sidebar, click Connectors.
2. Click Sources.
3. Cick New or Create Connector.
4. For Name, enter some unique name for this connector.
5. In the Provider area, click the source location type that matches yours.
6. Click Continue.
7. Fill in the fields with the appropriate settings. Learn more.
8. If a Continue button appears, click it, and fill in any additional settings fields.
9. Click Save and Test.

1. In the sidebar, click Connectors.
2. Click Destinations.
3. Cick New or Create Connector.
4. For Name, enter some unique name for this connector.
5. In the Provider area, click the destination location type that matches yours.
6. Click Continue.
7. Fill in the fields with the appropriate settings. Learn more.
8. If a Continue button appears, click it, and fill in any additional settings fields.
9. Click Save and Test.

1. In the sidebar, click Workflows.

2. Click New Workflow.

3. Next to Build it with me, click Create Workflow.

   If a radio button appears instead of Build it with me, select it, and then click Continue.

4. For Workflow Name, enter some unique name for this workflow.

5. In the Sources dropdown list, select your source location from Step 3.

6. In the Destinations dropdown list, select your destination location from Step 4.

   You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.

7. Click Continue.

8. In the Optimize for section, select the option to choose one of these preconfigured workflow settings groups:

   • Basic: Ideal for simple, text-only documents.

   • Advanced: Best for PDFs, images, and complex file types.

   • Platinum: For your most challenging documents, including scanned and handwritten content. It uses vision language models (VLMs). During processing, files that are not PDFs or images are processed by using the Advanced strategy and are charged at the Advanced rate instead.

     When you use the Platinum strategy for PDF files of 200 or more pages, you might notice some errors when these files are processed. These errors typically occur when these larger PDF files have lots of tables and high-resolution images.

9. The Reprocess all box applies only to the Amazon S3 and Azure Blob Storage source connectors:

   • Checking this box reprocesses all documents in the source location on every workflow run.
   • Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents’ contents change.

10. If you want to retry processing any documents that failed to process, check the Retry Failed Documents box.

11. Click Continue.

12. If you want this workflow to run on a schedule, in the Repeat Run dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select Don’t repeat.

13. Click Complete.

1. If you did not choose to run this workflow on a schedule in Step 5, you can run the workflow now: on the sidebar, click Workflows.
2. Next to your workflow from Step 5, click Run.

1. In the sidebar, click Jobs.
2. In the list of jobs, wait for the job’s Status to change to Finished.
3. Click the row for the job.
4. If Overview displays Success, go to the next Step.

Go to your destination location to view the processed data.

Go to the For Developers page and enter your information.

1. After you have signed up through the For Developers page, the Unstructured Platform sign-in page appears.

   If you signed up through the For Enterprise page instead, your sign-in process will be different. For enterprise sign-in guidance, contact Unstructured Sales at sales@unstructured.io.

2. Click Google or GitHub to sign in with the Google or GitHub account that you signed up with through the For Developers page. Or, enter the email address that you signed up with, and then click Sign In.

3. If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the Sign In link.

4. The first time you sign in, read the terms and conditions, and then click Accept.

1. At the bottom of the sidebar, click your user icon, and then click Account Settings.
2. On the account settings sidebar, click API Keys, if it is not already selected.
3. To get your API key, click the copy icon in the Actions column for your API key, and then click Key Only. Store your copied API key in a secure location. Do not share it with others.
4. To get your API URL, click the copy icon next to the URL next to API URL. Store your copied API URL in a secure location. Do not share it with others.

1. Set an environment variable named UNSTRUCTURED_API_KEY to the value of your Unstructured API key.
2. Set another environment variable named UNSTRUCTURED_API_URL to the value of your Unstructured API URL.

Run the following command:

pip install unstructured-ingest

You might also need to install additional dependencies, depending on your needs. Learn more.

Run the following code, replacing:

• <path/to/input> with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
• <path/to/output> with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

import os

from unstructured_ingest.v2.pipeline.pipeline import Pipeline
from unstructured_ingest.v2.interfaces import ProcessorConfig
from unstructured_ingest.v2.processes.connectors.local import (
    LocalIndexerConfig,
    LocalDownloaderConfig,
    LocalConnectionConfig,
    LocalUploaderConfig
)
from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

if __name__ == "__main__":
    Pipeline.from_configs(
        context=ProcessorConfig(),
        indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
        downloader_config=LocalDownloaderConfig(),
        source_connection_config=LocalConnectionConfig(),
        partitioner_config=PartitionerConfig(
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
            strategy="hi_res",
            additional_partition_args={
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        ),
        uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
    ).run()

Go to your destination location to view the processed data.