Azure Blob Storage

First time creating a connector? Read this first.

Ingest your files into Unstructured from Azure Blob Storage.

Requirements

You will need: The following video shows how to fulfill the minimum set of Azure Storage account requirements:

If you are generating an SAS token as shown in the preceding video, be sure to set the following permissions:

Read and List for reading from the container only.
Write and List for writing to the container only.
Read, Write, and List for both reading from and writing to the container.

Here are some more details about these requirements:

An Azure account. To create one, learn how.
An Azure Storage account, and a container within that account. Create a storage account. Create a container.
The Azure Storage remote URL, using the format az://<container-name>/<path/to/file/or/folder/in/container/as/needed> For example, if your container is named my-container, and there is a folder in the container named my-folder, the Azure Storage remote URL would be az://my-container/my-folder/.
An SAS token (recommended), access key, or connection string for the Azure Storage account. Create an SAS token (recommended). Get an access key. Get a connection string. Create an SAS token (recommended):
Get an access key or connection string:

Examples

To create an Azure Blob Storage source connector, see the following examples. For more information on working with source connectors using the Unstructured API, see Source endpoints.

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import CreateSourceConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type="azure",
                config={
                    "remote_url": "az://<container-name>/<path/to/file/or/folder>",
                    "recursive": <True|False>,

                    # For anonymous authentication, omit the following auth keys.
                    
                    # For SAS token authentication:
                    # "account_name": "<account-name>",
                    # "sas_token": "<sas-token>",

                    # For account key authentication:
                    # "account_name": "<account-name>",
                    # "account_key": "<account-key>",

                    # For connection string authentication:
                    # "connection_string": "<connection-string>"
                }
            )
        )
    )

    print(response.source_connector_information)

curl --request 'POST' --location \
"$UNSTRUCTURED_API_URL/sources" \
--header 'accept: application/json' \
--header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
--header 'content-type: application/json' \
--data \
'{
    "name": "<name>",
    "type": "azure",
    "config": {
        "remote_url": "az://<container-name>/<path/to/file/or/folder>",
        "recursive": <true|false>,
    
        # For anonymous authentication, do not set any of the 
        # following fields.

        # For SAS token authentication:
        "account_name": "<account-name>",
        "sas_token": "<sas-token>"

        # For account key authentication:
        "account_name": "<account-name>",
        "account_key": "<account-key>"

        # For connection string authentication:
        "connection_string": "<connection-string>"
    }
}'

Configuration settings

Replace the preceding placeholders as follows:

string

required

A unique name for this connector.

string

required

The Azure Storage remote URL, with the format az://<container-name>/<path/to/file/or/folder/in/container/as/needed>. For example, if your container is named my-container, and there is a folder in the container named my-folder, the Azure Storage remote URL would be az://my-container/my-folder/.

string

The Azure Storage account name. Required for SAS token authentication and account key authentication.

string

required

For SAS token authentication, the SAS token for the Azure Storage account.

string

required

For account key authentication, the key for the Azure Storage account.

string

required

For connection string authentication, the connection string for the Azure Storage account.

boolean

default:"false"

Source connector only. Set to true to recursively access files from subfolders within the container.

Set up Enterprise Connect authentication

Enterprise Connect is available for dedicated instance customers only, and must be enabled on your instance before use. Contact your Unstructured account team or Unstructured Support to request access and have it enabled.

Enterprise Connect is an authentication method for Azure connectors. It uses a federated identity credential to authenticate Unstructured as a customer-configured App Registration. During a workflow run, Unstructured uses this credential to receive a short-lived access token. Tokens expire automatically and no secrets are stored. For an overview, see Enterprise Connect for Azure. To configure an Azure Blob Storage connector to use Enterprise Connect, first complete the following setup in your Azure subscription:

Create an App Registration for Unstructured in Microsoft Entra ID. In your Azure subscription, follow the instructions in How to register an app in Microsoft Entra ID in the Microsoft Entra documentation. Enter a meaningful name for your App Registration (for example, unstructured-connector). For Supported account types, select Single tenant only. You are registering this app for a third-party service (Unstructured) accessing resources in your own tenant. This is the single-tenant scenario as defined by Microsoft.

Add a federated identity credential to the App Registration. Follow the instructions in Configure an app to trust an external identity provider in the Microsoft Entra documentation. Navigate to your App Registration, select Certificates & secrets in the left navigation pane, select the Federated credentials tab, and select Add credential. For Federated credential scenario, select Other issuer. Set the following values:

Field	Value
Name	A unique name for this credential (for example, `unstructured-federated-credential`). This cannot be changed after creation.
Issuer	The OIDC issuer URL for your Unstructured instance. Get this value from your Unstructured account team. (Example: `https://oidc.prod-aks.example.com/...`)
Subject	The service account identity for your Unstructured instance. Get this value from your Unstructured account team. (Example: `system:serviceaccount:etl:etl-job-runner`) This value must exactly match what Unstructured provides. If it does not match, the token exchange will fail without displaying an error.
Audience	Set this to `api://AzureADTokenExchange`.

Your Unstructured instance may require more than one federated identity credential. The platform uses separate identities for different operations, such as connection testing and running workflows. If your account team provides more than one Subject value, repeat these steps for each one.

Add a role assignment to grant your App Registration access to your Azure Blob Storage account. See Assign Azure roles using the Azure portal in the Azure documentation. Use the following values:
- Scope: the Azure Blob Storage account that contains the data you want the connector to access.
- Role: select Storage Blob Data Reader for a source, or Storage Blob Data Contributor for a destination.
- Members: select User, group, or service principal, then search for and select the App Registration you created in Step 1.
When you reach the Review + assign tab, click Review + assign to complete the assignment.
Note the following values from your App Registration. You will need them when configuring the connector in Unstructured. Both values are available on the Overview page of your App Registration in the Microsoft Entra admin center.
- The Tenant ID (also called Directory ID) for your Azure subscription.
- The Client ID of your App Registration.

Next, see the Create the connector with Enterprise Connect section below for examples.

Create the source connector with Enterprise Connect

The following examples create an Azure Blob Storage source connector using Enterprise Connect authentication. For more information on working with source connectors using the Unstructured API, see Source endpoints.

import os

from unstructured_client import UnstructuredClient
from unstructured_client.models.operations import CreateSourceRequest
from unstructured_client.models.shared import CreateSourceConnector

with UnstructuredClient(api_key_auth=os.getenv("UNSTRUCTURED_API_KEY")) as client:
    response = client.sources.create_source(
        request=CreateSourceRequest(
            create_source_connector=CreateSourceConnector(
                name="<name>",
                type="azure",
                config={
                    "remote_url": "az://<container-name>/<path/to/file/or/folder>",
                    "account_name": "<account-name>",
                    "tenant_id": "<tenant-id>",
                    "client_id": "<client-id>",
                    "recursive": <True|False>  # Boolean: True or False, no quotes
                }
            )
        )
    )

    print(response.source_connector_information)

curl --request 'POST' --location \
"$UNSTRUCTURED_API_URL/sources" \
--header 'accept: application/json' \
--header "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
--header 'content-type: application/json' \
--data \
'{
    "name": "<name>",
    "type": "azure",
    "config": {
        "remote_url": "az://<container-name>/<path/to/file/or/folder>",
        "account_name": "<account-name>",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "recursive": <true|false>
    }
}'

Replace the preceding placeholders as follows.

string

required

A unique name for this connector.

string

required

string

required

The Azure Storage account name.

string

required

The Tenant ID (also called Directory ID) for your Azure subscription.

string

required

The Client ID of your App Registration.

boolean

default:"false"

Source connector only. Set to true to recursively access files from subfolders within the container.

Unstructured API

Pipeline API

API Reference

Legacy APIs

Troubleshooting

Requirements

Examples

Configuration settings

Set up Enterprise Connect authentication

Create the source connector with Enterprise Connect

​Requirements

​Examples

​Configuration settings

​Set up Enterprise Connect authentication

​Create the source connector with Enterprise Connect

Requirements

Examples

Configuration settings

Set up Enterprise Connect authentication

Create the source connector with Enterprise Connect