Unstructured

To sign up for the Unstructured Platform, go to the For Developers page and choose one of the following plans:

• Sign up for a pay-per-page plan.
• Save money by signing up for a subscribe-and-save plan instead.

If you’re not sure which plan to sign up for, start with a pay-per-page plan. You can always switch plans later.

1. After you have signed up for a pay-per-page plan, the Unstructured Platform sign-in page appears.

2. Click Google or GitHub to sign in with the Google or GitHub account that you signed up with. Or, enter the email address that you signed up with, and then click Sign In.

3. If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the Sign In link.

4. The first time you sign in, read the terms and conditions, and then click Accept.

1. From your Unstructured Platform dashboard, in the sidebar, click Connectors.
2. Click Sources.
3. Cick New or Create Connector.
4. For Name, enter some unique name for this connector.
5. In the Provider area, click the source location type that matches yours.
6. Click Continue.
7. Fill in the fields with the appropriate settings. Learn more.
8. If a Continue button appears, click it, and fill in any additional settings fields.
9. Click Save and Test.

1. In the sidebar, click Connectors.
2. Click Destinations.
3. Cick New or Create Connector.
4. For Name, enter some unique name for this connector.
5. In the Provider area, click the destination location type that matches yours.
6. Click Continue.
7. Fill in the fields with the appropriate settings. Learn more.
8. If a Continue button appears, click it, and fill in any additional settings fields.
9. Click Save and Test.

1. In the sidebar, click Workflows.

2. Click New Workflow.

3. Next to Build it for Me, click Create Workflow.

   If a radio button appears instead of Build it for Me, select it, and then click Continue.

4. For Workflow Name, enter some unique name for this workflow.

5. In the Sources dropdown list, select your source location from Step 3.

6. In the Destinations dropdown list, select your destination location from Step 4.

   You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.

7. Click Continue.

8. The Reprocess All box applies only to blob storage connectors such as the Amazon S3, Azure Blob Storage, and Google Cloud Storage connectors:

   • Checking this box reprocesses all documents in the source location on every workflow run.
   • Unchecking this box causes new documents that have been added to the source location, as well as existing documents in the source location that have had their contents or titles changed, since the last workflow run to be processed on future runs. Other previously processed documents are not processed again.

9. Click Continue.

10. If you want this workflow to run on a schedule, in the Repeat Run dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select Don’t repeat.

11. Click Complete.

1. If you did not choose to run this workflow on a schedule in Step 5, you can run the workflow now: on the sidebar, click Workflows.
2. Next to your workflow from Step 5, click Run.

1. In the sidebar, click Jobs.
2. In the list of jobs, wait for the job’s Status to change to Finished.
3. Click the row for the job.
4. After Overview displays Finished, go to the next Step.

Go to your destination location to view the processed data.

Sign up for an Unstructured account by going to the For Developers page and choosing one of the following plans:

• Sign up for a pay-per-page plan.
• Save money by signing up for a subscribe-and-save plan instead.

If you’re not sure which plan to sign up for, start with a pay-per-page plan. You can always switch plans later.

1. After you have signed up through the For Developers page, the Unstructured account sign-in page appears.

   If you signed up through the For Enterprise page instead, your sign-in process will be different. For enterprise sign-in guidance, contact Unstructured Sales at sales@unstructured.io.

2. Click Google or GitHub to sign in with the Google or GitHub account that you signed up with through the For Developers page. Or, enter the email address that you signed up with, and then click Sign In.

3. If you entered your email address, check your email inbox for a message from Unstructured. In that email, click the Sign In link.

4. The first time you sign in, read the terms and conditions, and then click Accept.

1. At the bottom of the sidebar, click your user icon, and then click Account Settings.
2. Click the API Keys tab, if it is not already selected.
3. To get your API key, click the copy icon in the Actions column for your API key, and then click Key Only. Store your copied API key in a secure location. Do not share it with others.
4. To get your API URL, click the copy icon next to the URL next to API URL. Store your copied API URL in a secure location. Do not share it with others.

1. Set an environment variable named UNSTRUCTURED_API_KEY to the value of your Unstructured API key.
2. Set another environment variable named UNSTRUCTURED_API_URL to the value of your Unstructured API URL.

Run the following command:

pip install unstructured-ingest

You might also need to install additional dependencies, depending on your needs. Learn more.

Run the following code, replacing:

• <path/to/input> with the source (input) path to the directory on your local machine that contains the compatible files for Unstructured to process on its hosted compute resources.
• <path/to/output> with the destination (output) path to the directory on your local machine that will contain the processed data that Unstructured returns from its hosted compute resources.

import os

from unstructured_ingest.v2.pipeline.pipeline import Pipeline
from unstructured_ingest.v2.interfaces import ProcessorConfig
from unstructured_ingest.v2.processes.connectors.local import (
    LocalIndexerConfig,
    LocalDownloaderConfig,
    LocalConnectionConfig,
    LocalUploaderConfig
)
from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

if __name__ == "__main__":
    Pipeline.from_configs(
        context=ProcessorConfig(),
        indexer_config=LocalIndexerConfig(input_path=os.getenv("LOCAL_FILE_INPUT_DIR")),
        downloader_config=LocalDownloaderConfig(),
        source_connection_config=LocalConnectionConfig(),
        partitioner_config=PartitionerConfig(
            partition_by_api=True,
            api_key=os.getenv("UNSTRUCTURED_API_KEY"),
            partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
            strategy="hi_res",
            additional_partition_args={
                "split_pdf_page": True,
                "split_pdf_allow_failed": True,
                "split_pdf_concurrency_level": 15
            }
        ),
        uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
    ).run()

Go to your destination location to view the processed data.