Azure Cognitive Search
Batch process all your records to store structured outputs in an Azure Cognitive Search account.
You will need:
The Azure AI Search (formerly Azure Cognitive Search) prerequisites:
The following video shows how to fulfill the minimum set of Azure AI Search prerequisites:
Here are some more details about these prerequisites:
-
The endpoint and API key for Azure AI Search. Create an endpoint and API key.
-
The name of the index in Azure AI Search. Create an index.
The Azure AI Search index that you use must have an index schema that is compatible with the schema of the documents that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all circumstances. This is because these schemas will vary based on your source files’ types; how you want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.
You can adapt the following index schema example for your own needs:
{ "name": "<your-index-name>", "fields": [ { "name": "id", "type": "Edm.String", "key": true, "retrievable": true }, { "name": "element_id", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": true, "facetable": false }, { "name": "type", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": true, "facetable": true }, { "name": "text", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false }, { "name": "embeddings", "type": "Collection(Edm.Single)", "searchable": true, "filterable": false, "sortable": false, "facetable": false, "dimensions": 384, "vectorSearchProfile": "embeddings-config" }, { "name": "metadata", "type": "Edm.ComplexType", "fields": [ { "name": "parent_id", "type": "Edm.String", "searchable": false, "filterable": true, "sortable": true, "facetable": false }, { "name": "page_number", "type": "Edm.Int32", "searchable": false, "filterable": true, "sortable": true, "facetable": true }, { "name": "is_continuation", "type": "Edm.Boolean", "searchable": false, "filterable": true, "sortable": true, "facetable": true }, { "name": "orig_elements", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false } ] } ], "vectorSearch": { "compressions": [ { "name": "scalar-quantization", "kind": "scalarQuantization", "rerankWithOriginalVectors": true, "defaultOversampling": 10.0, "scalarQuantizationParameters": { "quantizedDataType": "int8" } } ], "algorithms": [ { "name": "hnsw-1", "kind": "hnsw", "hnswParameters": { "metric": "cosine", "m": 4, "efConstruction": 400, "efSearch": 500 } } ], "profiles": [ { "name": "embeddings-config", "algorithm": "hnsw-1", "compression": "scalar-quantization" } ] } }
See also:
The Azure Cognitive Search connector dependencies:
pip install "unstructured-ingest[azure-cognitive-search]"
You might also need to install additional dependencies, depending on your needs. Learn more.
These environment variables:
AZURE_SEARCH_ENDPOINT
- The endpoint URL for Azure AI Search (formerly Azure Cognitive Search), represented by--endpoint
(CLI) orendpoint
(Python).AZURE_SEARCH_API_KEY
- The API key for Azure AI Search, represented by--key
(CLI) orkey
(Python).AZURE_SEARCH_INDEX
- The name of the index for Azure AI Search, represented by--index
(CLI) orindex
(Python).
Now call the Unstructured CLI or Python. The source connector can be any of the ones supported. This example uses the local source connector.
This example sends files to Unstructured API services for processing by default. To process files locally instead, see the instructions at the end of this page.
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the --partition-by-api
option (CLI) or partition_by_api
(Python) parameter to specify where files are processed:
-
To do local file processing, omit
--partition-by-api
(CLI) orpartition_by_api
(Python), or explicitly specifypartition_by_api=False
(Python).Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
--api-key $UNSTRUCTURED_API_KEY
(CLI) orapi_key=os.getenv("UNSTRUCTURED_API_KEY")
(Python)--partition-endpoint $UNSTRUCTURED_API_URL
(CLI) orpartition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
(Python)- The environment variables
UNSTRUCTURED_API_KEY
andUNSTRUCTURED_API_URL
-
To send files to Unstructured API services for processing, specify
--partition-by-api
(CLI) orpartition_by_api=True
(Python).Unstructured API services also requires an Unstructured API key and API URL, by adding the following:
--api-key $UNSTRUCTURED_API_KEY
(CLI) orapi_key=os.getenv("UNSTRUCTURED_API_KEY")
(Python)--partition-endpoint $UNSTRUCTURED_API_URL
(CLI) orpartition_endpoint=os.getenv("UNSTRUCTURED_API_URL")
(Python)- The environment variables
UNSTRUCTURED_API_KEY
andUNSTRUCTURED_API_URL
, representing your API key and API URL, respectively.