Workflows

​

Workflows dashboard

To view the workflows dashboard, on the sidebar, click Workflows.

A workflow in the Unstructured Platform is a defined sequence of processes that automate the data handling from source to destination. It allows users to configure how and when data should be ingested, processed, and stored.

Workflows are crucial for establishing a systematic approach to managing data flows within the platform, ensuring consistency, efficiency, and adherence to specific data processing requirements.

​

Create a workflow

The Unstructured Platform provides two types of workflow builders:

• Automatic workflows, which use sensible default workflow settings to enable you to get good-quality results faster.
• Custom workflows, which enable you to fine-tune the workflow settings behind the scenes to get very specific results.

​

Create an automatic workflow

To create an automatic workflow:

1. On the sidebar, click Workflows.

2. Click New Workflow.

3. Next to Build it with Me, click Create Workflow.

   If a radio button appears instead of Build it with Me, select it, and then click Continue.

4. For Workflow Name, enter some unique name for this workflow.

5. In the Sources dropdown list, select your source location.

6. In the Destinations dropdown list, select your destination location.

   You can select multiple source and destination locations. Files will be ingested from all of the selected source locations, and the processed data will be delivered to all of the selected destination locations.

7. Click Continue.

8. In the Optimize for section, select the option to choose one of these preconfigured workflow settings groups. Expand any or all of the following options to learn more about these preconfigured settings:

   Basic

   This option is ideal for simple, text-only documents.

   The Basic option uses the following preconfigured workflow settings:

   • Strategy: Fast

   • Image Summarizer: None

   • Table Summarizer: None

   • Include Page Breaks: No

   • Infer Table Structure: No

   • Elements to Exclude: None

   • Chunk:

     • Chunker Type: Chunk By Character

     • Chunk Options:

       • Include Original Elements: No
       • Max Characters: 2048
       • New After N Characters: 1500
       • Overlap: 160
       • Overlap All: No

   • Embed:

     • Provider: Azure OpenAI
     • Model: text-embedding-3-large (3072 dimensions)

   Advanced

   This option is best for PDFs, images, and complex file types.

   During Advanced processing, any detected text-based files are processed and billed at the Basic rate instead.

   The Advanced option uses the following preconfigured workflow settings:

   • Strategy: High-Res

   • Image Summarizer: GPT-4o

   • Table Summarizer: Claude 3.5 Sonnet

   • Include Page Breaks: No

   • Infer Table Structure: No

   • Elements to Exclude: None

   • Chunk:

     • Chunker Type: Chunk By Title

     • Chunk Options:

       • Combine Text Under N Characters: 0
       • Include Original Elements: No
       • Max Characters: 2048
       • New After N Characters: 1500
       • Overlap: 160
       • Overlap All: No

   • Embed:

     • Provider: Azure OpenAI
     • Model: text-embedding-3-large (3072 dimensions)

   Platinum

   This option is for your most challenging documents, including scanned and handwritten content.

   During Platinum processing, any detected files that are not PDFs or images are processed and billed at either the Advanced or Basic rate instead. Of those non-PDF and non-image files, all text-based files are processed and billed at the Basic rate instead. The other files are processed and billed at the Advanced rate instead.

   When you use the Platinum strategy for PDF files of 200 or more pages, you might notice some errors when these files are processed. These errors typically occur when these larger PDF files have lots of tables and high-resolution images.

   The Platinum option uses the following preconfigured workflow settings:

   • Strategy: VLM

   • VLM Provider, Model: Anthropic, Anthropic Claude 3.5 Sonnet

   • Chunk:

     • Chunker Type: Chunk By Title

     • Chunk Options:

       • Combine Text Under N Characters: 0
       • Include Original Elements: No
       • Max Characters: 2048 = Multipage Sections: No
       • New After N Characters: 1500
       • Overlap: 160
       • Overlap All: No

   • Embed:

     • Provider: Azure OpenAI
     • Model: text-embedding-3-large (3072 dimensions)

9. The Reprocess All box applies only to blob storage connectors such as the Amazon S3, Azure Blob Storage, and Google Cloud Storage connectors:

   • Checking this box reprocesses all documents in the source location on every workflow run.
   • Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents’ contents change.

10. Click Continue.

11. If you want this workflow to run on a schedule, in the Repeat Run dropdown list, select one of the scheduling options, and fill in the scheduling settings. Otherwise, select Don’t repeat.

12. Click Complete.

13. If you did not set the workflow to run on a schedule, you can run the worklow now.

​

Create a custom workflow

1. On the sidebar, click Workflows.

2. Click New Workflow.

3. Click the Build it Myself option, and then click Continue.

4. In the This workflow pane, click the Details button.

5. Next to Name, click the pencil icon, enter some unique name for this workflow, and then click the check mark icon.

6. If you want this workflow to run on a schedule, click the Schedule button. In the Repeat Run dropdown list, select one of the scheduling options, and fill in the scheduling settings.

7. To overwrite any previously processed files, or to retry any documents that fail to process, click the Settings button, and check either or both of the boxes.

   The Reprocess All box applies only to blob storage connectors such as the Amazon S3, Azure Blob Storage, and Google Cloud Storage connectors:

   • Checking this box reprocesses all documents in the source location on every workflow run.
   • Unchecking this box causes new documents that have been added to the source location, as well as existing documents in the source location that have had their contents or titles changed, since the last workflow run to be processed on future runs. Other previously processed documents are not processed again.

8. The workflow begins with the following layout:

   The following workflow layouts are also valid:

9. In the pipeline designer, click the Source node. In the Source pane, select the source location. Then click Save.

10. Click the Destination node. In the Destination pane, select the destination location. Then click Save.

11. As needed, add more nodes by clicking the plus icon (recommended) or Add Node button:

    • Click Connect to add another Source or Destination node. You can add multiple source and destination locations. Files will be ingested from all of the source locations, and the processed data will be delivered to all of the destination locations. Learn more.
    • Click Enrich to add a Chunker or Enrichment node. Learn more.
    • Click Transform to add a Partitioner or Embedder node. Learn more.
    Make sure to add nodes in the correct order. If you are unsure, see the usage hints in the blue note that appears in the node’s settings pane.

    To edit a node, click that node, and then change its settings.

    To delete a node, click that node, and then click the trash can icon above it.

12. Click Save.

13. If you did not set the workflow to run on a schedule, you can run the worklow now.

​

Custom workflow node types

​

Edit, delete, or run a workflow

To run a workflow once, manually:

1. On the sidebar, click Workflows.
2. In the list of workflows, click Run in the row for the workflow that you want to run.

For each of the workflows on the Workflows list page, the following actions are available by clicking the ellipses (the three dots) in the row for the respective workflow:

• Edit via Form: Changes the existing configuration of your workflow.
• Delete: Removes the workflow from the platform. Use this action cautiously, as it will permanently delete the workflow and its configurations.
• Open: Opens the workflow’s settings page.

​

Pause a scheduled workflow

To stop running a workflow that is set to run on a repeating schedule:

1. On the sidebar, click Workflows.
2. In the list of workflows, turn off the Status toggle in the row for the workflow that you want to stop running on a repeated schedule.

Turning off the Status toggle also disables the workflow’s Run button, which prevents that workflow from being run manually as well.

To resume running the workflow on its original repeating schedule, as well as enable the workflow to be run manually as needed, turn on the workflow’s Status toggle.