Jobs

A workflow job is created automatically whenever:

A workflow runs on a schedule. For more details see Create workflow.
You run a workflow manually. For more details see Run workflow.

To use the Unstructured Pipelines operations to manage jobs, do the following:

Task	Python function	curl
Get a list of available jobs	`jobs.list_jobs`	`GET` `/jobs`
Get basic information about a job	`jobs.get_job`	`GET` `/jobs/<job-id>`
Get information about a job’s current processing status	`jobs.get_job_details`	`GET` `/jobs/<job-id>/details`
Get the list of any failed files for a job and why those files failed	`jobs.get_failed_files`	`GET` `/jobs/<job-id>/failed-files`
Run a workflow that takes one or more local files only as input, and the workflow exists only for the duration of that job’s run		`POST` `/jobs/`
Cancel a running job	`jobs.cancel_job`	`POST` `/jobs/<job-id>/cancel`

Preflight check

The Job Preflight Check runs before a job processes any documents. It checks that the credentials and settings are valid for each source connector, destination connector, and model-provider node (partitioner, chunker, embedder, enrichment, and extractor). It does not read or write documents. The check runs while the job is Pending (SCHEDULED in the API). Most checks finish in seconds, and some take a few minutes. You can cancel a pending job.

Which check results reject a job

If any node fails the check with one of these results, the job ends as Rejected (REJECTED in the API):

Result	Label in Unstructured Pipelines	Meaning
`AUTH_INVALID_CREDENTIAL`	Invalid credentials	The provider did not accept the credentials.
`AUTH_PERMISSION_DENIED`	Permission denied	The credentials lack permission for the requested resource.
`CONFIG_INVALID`	Invalid configuration	A connector or node setting is not valid.

Any other result lets the job run: AUTH_NETWORK_UNREACHABLE, AUTH_PROVIDER_UNAVAILABLE, PROVIDER_RATE_LIMITED, UNKNOWN, or a check that does not finish in time. If the problem is real, the job can still end as Failed (FAILED). A rejected job processed no documents. Rejected is a final status, and Unstructured does not retry the job.

Jobs that skip the preflight check

Test runs on local files that you upload in Unstructured Pipelines, such as on the Start page.
Jobs created from a workflow template.
Jobs for a workflow with the check turned off. Select Skip Preflight Check in the workflow settings, or set skip_preflight to true when you create or update a workflow. For a local-file processing job, set skip_preflight in the request_data of the create job request.

Without the check, a misconfigured connector fails partway through the run, and the job ends as Failed.

On dedicated instance and in-VPC deployments, availability of the Job Preflight Check depends on your deployment. Contact your Unstructured account team for details.

Find why a job was rejected

A rejected job has the status REJECTED, and its reason on the get job endpoint is null. The cause is on the job’s nodes:

Call the get job details endpoint.
In node_stats, find each node whose failure_category is not null. Act on failure_category. Show failure_reason to a person, but do not parse it, because its text can change.
Fix the connector or node, and then run the workflow again. Each run creates a new job.

Canceling a rejected job returns HTTP 409 with the detail Job already rejected. Python SDK users need unstructured-client 0.47.0 or later. Earlier versions raise ResponseValidationError on any REJECTED job, including on every page of list_jobs that contains one. This script polls a job and, if the job is rejected, prints each rejected node’s category and reason. Replace <job-id> with your job ID. The script requires jq.

#!/usr/bin/env bash

JOB_ID="<job-id>"

while true; do
    status=$(curl --silent --request GET \
      --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/${JOB_ID}" \
      --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" | jq -r '.status')
    echo "Job status: $status"

    case "$status" in
        COMPLETED|STOPPED|FAILED) break ;;
        REJECTED)
            curl --silent --request GET \
              --url "${UNSTRUCTURED_API_URL}/api/v1/jobs/${JOB_ID}/details" \
              --header "unstructured-api-key: ${UNSTRUCTURED_API_KEY}" \
            | jq -r '.node_stats[]
                | select(.failure_category != null)
                | "\(.node_name): \(.failure_category)\n  \(.failure_reason)"'
            exit 1 ;;
    esac

    sleep 10
done

Example output:

Job status: SCHEDULED
Job status: REJECTED
S3 Source: AUTH_INVALID_CREDENTIAL
  [AUTH_INVALID_CREDENTIAL] Connection test failed.
S3 authentication failed: InvalidAccessKeyId
S3 Destination: AUTH_INVALID_CREDENTIAL
  [AUTH_INVALID_CREDENTIAL] Connection test failed.
S3 authentication failed: InvalidAccessKeyId

For fixes by category, see Job rejected by the preflight check.

Visit the Unstructured documentation samples repo on GitHub for Postman Collections that demonstrate the API requests discussed in this topic. To import any of these Postman collections:

Browse to the postman folder in the repo, and select the collection file you want to import.
Click the Raw icon, and then copy the URL for the raw JSON file.
In your Postman workspace, click Import, and then paste in the file URL.

To learn about Postman Collections, see Manage and organize Postman Collections in the Postman Docs.

Developer guide

API reference

Troubleshooting

Preflight check

Which check results reject a job

Jobs that skip the preflight check

Find why a job was rejected

​Preflight check

​Which check results reject a job

​Jobs that skip the preflight check

​Find why a job was rejected

Preflight check

Which check results reject a job

Jobs that skip the preflight check

Find why a job was rejected