> ## Documentation Index
> Fetch the complete documentation index at: https://docs.unstructured.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Box
Connect Box to your preprocessing pipeline, and use the Unstructured CLI or Python to batch process all your documents and store structured outputs locally on your filesystem.
The requirements are as follows.
1. Access to the [Developer Console](https://app.box.com/developers/console) from your [Box enterprise account](https://account.box.com/signup/enterprise-plan) or [Box developer account](https://account.box.com/signup/developer).
2. A Box Custom App in your Box account, set up to use **Server Authentication (with JWT)**. See [Setup with JWT](https://developer.box.com/guides/authentication/jwt/jwt-setup/).
3. The appropriate application scopes and advanced features set up for the Box Custom App, as follows:
* In the Box Custom App, on the **Configuration** tab, under **Application Scopes**, check the box titled **Write all files and folders stored in Box**.
* In the Box Custom App, on the **Configuration** tab, under **Advanced Features**, check the box titled **Make API calls using the as-user header**.
* Click **Save Changes** before continuing.
4. Authorization by a Box Admin in your Box account for the Box Custom App. See [App Authorization](https://developer.box.com/guides/authentication/jwt/jwt-setup/#app-authorization).
5. Access by your Box account's source or target [folder](https://app.box.com/folder/0) to your Box Custom App, as follows:
* In the Box Custom App, on the **General Settings** tab, copy the **Service Account ID** (which takes the form `AutomationUser__`.
6. The private key configuration JSON file for the Box Custom App, or a string that contains this file's contents.
* To download this file, in the Box Custom App, on the **Configuration** tab, under **Add and Manage Public Keys**, click **Generate a Public/Private Keypair**. Store the downloaded private key configuration JSON file in a secure location.
* To ensure maximum compatibility across Unstructured service offerings, you should give the private key configuration JSON file information to Unstructured as
a single-line string that contains the contents of the downloaded private key configuration JSON file (and not the file itself).
To print this single-line string, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt.
In this command, replace `` with the path to the private key configuration JSON file that you downloaded by following the preceding instructions.
* For macOS or Linux:
```text theme={null}
tr -d '\n' <
```
* For Windows:
```text theme={null}
(Get-Content -Path "" -Raw).Replace("`r`n", "").Replace("`n", "")
```
## Document permissions metadata
The source connector outputs any permissions information that it can find in the source location about the processed source files, and associates that information with each corresponding element that is generated. This permissions information is output into the `permissions_data` field, which is within the
`data_source` field under the element's `metadata` field (`metadata.data_source.permissions_data`). This information lists the users or groups, if any, that have
permissions to read, update, or delete the element's associated source document.
The permissions metadata Unstructured outputs should not be used for runtime authorization
or access control enforcement.
Unstructured outputs document permissions metadata that is accurate only
at the point in time when Unstructured ingested the corresponding document
to which those permissions applied. Because this metadata is a
point-in-time copy of the permissions in the source location, these
metadata outputs that are sent to your destination location are not
always guaranteed to match the current permissions in the source location.
Also, be aware that Unstructured updates permission metadata for a document only when the document's content has changed.
This is because Unstructured performs incremental processing of documents only when documents' *content* has changed—not when only the documents'
*permissions* have changed. Whenever Unstructured performs incremental processing of documents for a
workflow (in other words, if **Reprocess All Files** is turned off or set
to false for a workflow), that worfklow will not output metadata for any
document permissions that have been added, changed, or removed since the
previous workflow run, *unless* the corresponding documents' content
has also been changed since the previous workflow run.
Permissions are defined in Box through *collaborations*, which function similar to access control lists. Collaboration objects are attached to folders, and those collaborations cascade down to everything inside that folder. A file's effective permissions come from the chain of ancestor folders above it. For more information, see [Collaborations overview](https://developer.box.com/guides/collaborations/) in the Box developer documentation.
To determine a file's effective permissions, Unstructured traverses the full folder hierarchy for each file, collecting collaborations at every level, and merging them. It also includes collaborations applied directly to the file itself. For more information, see [List folder collaborations](https://developer.box.com/reference/get-folders-id-collaborations/) and [List file collaborations](https://developer.box.com/reference/get-files-id-collaborations/) in the Box developer documentation.
When Unstructured compiles a file's effective permissions:
* If the same user or group appears at multiple levels with different roles, Unstructured chooses the least-restrictive role.
* Unstructured does not include the following in the permission metadata. No sentinel value is written into the permission metadata.
* `all_users_group`: Box's built-in group representing every user in the enterprise.
* `is_access_only`: Collaboration property that determines whether, for collaborators, to display the items in the All Files list and let them see the path to the root folder for the shared item.
* Unstructured only includes collaborations with a status of `accepted`. It does not include collaborations with a status of `pending` or `rejected`.
Unstructured includes a maximum of 500 permissions per file.
Unstructured maps Box roles to `read`, `update`, and `delete` access:
| Box role | Box access | Unstructured permission metadata |
| -------------------- | -------------------------------------------------------- | -------------------------------- |
| `owner` | Full access | `read`, `update`, `delete` |
| `co-owner` | Full access except transfer ownership | `read`, `update`, `delete` |
| `editor` | View, download, upload, edit, delete, copy, move, rename | `read`, `update` |
| `viewer` | Preview, download, comment | `read` |
| `previewer` | Preview | `read` |
| `viewer uploader` | Preview, download, upload | `read` |
| `previewer uploader` | Preview, upload | `read` |
| `uploader` | Upload and see item names | excluded entirely |
Note the following:
* Unstructured excludes the `uploader` role because it confers no `read`, `update`, or `delete` access.
* Unstructured maps the Box `editor` role to `read` and `update` access but not `delete`, even though that role does allow deleting files.
For more information, see [Understanding collaborator permission level](https://support.box.com/hc/en-us/articles/360044196413-Understanding-Collaborator-Permission-Levels) at Box Support.
### Identifier format
Users and groups are identified by the numeric ID that Box assigns them. To retrieve information about a specific user or group, use appropriate Box API with the corresponding ID:
* [Get user](https://developer.box.com/reference/get-users-id/)
* [Get group](https://developer.box.com/reference/get-groups-id/)
### Metadata output example
The following example shows what the output looks like. Ellipses indicate content that has been omitted from this example for brevity.
```json theme={null}
[
{
"...": "...",
"metadata": {
"...": "...",
"data_source": {
"...": "...",
"permissions_data": [
{
"read": {
"users": ["11446688", "11336699"],
"groups": ["11223344"]
}
},
{
"update": {
"users": ["11446688"],
"groups": []
}
},
{
"delete": {
"users": [],
"groups": []
}
}
],
"...": "..."
}
}
}
]
```
The Box connector dependencies:
```bash CLI, Python theme={null}
pip install "unstructured-ingest[box]"
```
You might also need to install additional dependencies, depending on your needs. [Learn more](/open-source/ingestion/ingest-dependencies).
The following environment variables:
* `BOX_APP_CONFIG` - The local path to the downloaded private key configuration JSON file for the Box Custom App,
or a single-line string that contains the contents of this file, represented by `--box-app-config` (CLI) or `box_app_config` (Python).
* `BOX_REMOTE_URL` - The remote URL to the target folder, represented by `--remote-url` (CLI) or `remote_url` (Python).
This URL must take the format `box://`.
Now call the Unstructured CLI or Python. The destination connector can be any of the ones supported. This example uses the local destination connector.
This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.
```bash CLI theme={null}
#!/usr/bin/env bash
unstructured-ingest \
box \
--box-app-config $BOX_APP_CONFIG \
--remote-url $BOX_REMOTE_URL \
--output-dir $LOCAL_FILE_OUTPUT_DIR \
--num-processes 2 \
--recursive \
--verbose \
--partition-by-api \
--partition-endpoint $UNSTRUCTURED_API_URL \
--api-key $UNSTRUCTURED_API_KEY \
--strategy hi_res \
--additional-partition-args="{\"split_pdf_page\":\"true\", \"split_pdf_allow_failed\":\"true\", \"split_pdf_concurrency_level\": 15}" \
```
```python Python Ingest theme={null}
import os
from unstructured_ingest.pipeline.pipeline import Pipeline
from unstructured_ingest.interfaces import ProcessorConfig
from unstructured_ingest.processes.connectors.fsspec.box import (
BoxAccessConfig,
BoxConnectionConfig,
BoxIndexerConfig,
BoxDownloaderConfig
)
from unstructured_ingest.processes.partitioner import PartitionerConfig
from unstructured_ingest.processes.chunker import ChunkerConfig
from unstructured_ingest.processes.embedder import EmbedderConfig
from unstructured_ingest.processes.connectors.local import LocalUploaderConfig
# Chunking and embedding are optional.
if __name__ == "__main__":
Pipeline.from_configs(
context=ProcessorConfig(),
indexer_config=BoxIndexerConfig(remote_url=os.getenv("BOX_REMOTE_URL")),
downloader_config=BoxDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
source_connection_config=BoxConnectionConfig(
access_config=BoxAccessConfig(
box_app_config=os.getenv("BOX_APP_CONFIG")
)
),
partitioner_config=PartitionerConfig(
partition_by_api=True,
api_key=os.getenv("UNSTRUCTURED_API_KEY"),
partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
strategy="hi_res",
additional_partition_args={
"split_pdf_page": True,
"split_pdf_allow_failed": True,
"split_pdf_concurrency_level": 15
}
),
chunker_config=ChunkerConfig(chunking_strategy="by_title"),
embedder_config=EmbedderConfig(embedding_provider="huggingface"),
uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
).run()
```
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:
* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).
Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
* `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
* `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
* The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`
* To send files to the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).
Unstructured also requires an Unstructured API key and API URL, by adding the following:
* `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
* `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
* The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.
You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, which applies to **Let's Go**, **Pay-As-You-Go**, and **Business SaaS** accounts.
The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the legacy [Unstructured Partition Endpoint](/api-reference/legacy-api/partition/overview). However, you should always use the URL that was provided to you when your Unstructured account was created. If you do not have this URL, email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).
If you do not have an API key, [get one now](/api-reference/legacy-api/partition/overview).
If you are using a **Business** account, the process
for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
For instructions, see your Unstructured account administrator, or email Unstructured Support at [support@unstructured.io](mailto:support@unstructured.io).