Batch process all your records to store structured outputs in a Box account.

The requirements are as follows.

1. Access to the Developer Console from your Box enterprise account or Box developer account.

2. A Box Custom App in your Box account, set up to use Server Authentication (with JWT). See Setup with JWT.

3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:

   • In the Box Custom App, on the Configuration tab, under Application Scopes, check the box titled Write all files and folders stored in Box.
   • In the Box Custom App, on the Configuration tab, under Advanced Features, check the box titled Make API calls using the as-user header.
   • Click Save Changes before continuing.

4. Authorization by a Box Admin in your Box account for the Box Custom App. See App Authorization.

5. Access by your Box account’s source or target folder to your Box Custom App, as follows:

   • In the Box Custom App, on the General Settings tab, copy the Service Account ID (which takes the form AutomationUser_<your-app-service-id>_<a-random-string@boxdevedition.com).
   • Share your Box account’s target folder with the copied service account’s email address as a Co-owner or Editor.
   • Note the remote URL to the target folder, which takes the format box://<path/to/folder/in/account>.

6. The private key configuration JSON file for the Box Custom App, or a string that contains this file’s contents.

   • To download this file, in the Box Custom App, on the Configuration tab, under Add and Manage Public Keys, click Generate a Public/Private Keypair. Store the downloaded private key configuration JSON file in a secure location.

   • To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself). To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt. In this command, replace <path-to-downloaded-key-file> with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.

     • For macOS or Linux:

       Copy

       Ask AI

       tr -d '\n' < <path-to-downloaded-key-file>
       

     • For Windows:

       Copy

       Ask AI

       (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
       

The Box connector dependencies:

pip install "unstructured-ingest[box]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

• BOX_APP_CONFIG - The local path to the downloaded private key configuration JSON file for the Box Custom App, or a single-line string that contains the contents of this file, represented by --box-app-config (CLI) or box_app_config (Python).
• BOX_REMOTE_URL - The remote URL to the target folder, represented by --remote-url (CLI) or remote_url (Python). This URL must take the format box://<path/to/folder/in/account>.

Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.

This example sends files to Unstructured for processing by default. To process files locally instead, see the instructions at the end of this page.

#!/usr/bin/env bash

# Chunking and embedding are optional.

unstructured-ingest \
  local \
    --input-path $LOCAL_FILE_INPUT_DIR \
    --output-dir $LOCAL_FILE_OUTPUT_DIR \
    --strategy hi_res \
    --chunk-elements \
    --embedding-provider huggingface \
    --num-processes 2 \
    --verbose \
    --additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
  box \
    --box-app-config $BOX_APP_CONFIG \
    --remote-url $BOX_REMOTE_URL

For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api option (CLI) or partition_by_api (Python) parameter to specify where files are processed:

• To do local file processing, omit --partition-by-api (CLI) or partition_by_api (Python), or explicitly specify partition_by_api=False (Python).

  Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:

  • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
  • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
  • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL

• To send files to the Unstructured Partition Endpoint for processing, specify --partition-by-api (CLI) or partition_by_api=True (Python).

  Unstructured also requires an Unstructured API key and API URL, by adding the following:

  • --api-key $UNSTRUCTURED_API_KEY (CLI) or api_key=os.getenv("UNSTRUCTURED_API_KEY") (Python)
  • --partition-endpoint $UNSTRUCTURED_API_URL (CLI) or partition_endpoint=os.getenv("UNSTRUCTURED_API_URL") (Python)
  • The environment variables UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL, representing your API key and API URL, respectively.
  You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, for example, if you are using a self-hosted instance of the Unstructured API.

  The default API URL for Unstructured Ingest is https://api.unstructuredapp.io/general/v0/general, which is the API URL for the Unstructured Partition Endpoint. However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, contact Unstructured Sales at sales@unstructured.io.

  If you do not have an API key, get one now.

  If the Unstructured API is self-hosted, the process for generating Unstructured API keys, and the Unstructured API URL that you use, are different. For details, contact Unstructured Sales at sales@unstructured.io.