OpenSearch

Batch process all your records to store structured outputs in OpenSearch.

The requirements are as follows.

• An OpenSearch instance, such as an AWS OpenSearch instance…

  …or a local instance.

  In all cases, the OpenSearch version must be compatible with the client version of the opensearch-py package on PyPI.

• The instance’s hostname and port number. To learn how, see:

  • Creating and managing Amazon OpenSearch Service domains for AWS OpenSearch instances.
  • Communicate with OpenSearch for local OpenSearch instances.

• The name of the search index on the instance. Create an index.

  The OpenSearch index that you use must have a schema that is compatible with the schema of the documents that Unstructured produces for you. Unstructured cannot provide a schema that is guaranteed to work in all circumstances. This is because these schemas will vary based on your source files’ types; how you want Unstructured to partition, chunk, and generate embeddings; any custom post-processing code that you run; and other factors.

  You can adapt the following index schema example for your own needs:

  {
      "settings": {
          "index": {
              "knn": true,
              "knn.algo_param.ef_search": 100
          }
      },
      "mappings": {
          "properties": {
              "record_id": {
                  "type": "text"
              },
              "element_id": {
                  "type": "keyword"
              },
              "text": {
                  "type": "text"
              },
              "embeddings": {
                  "type": "knn_vector",
                  "dimension": 384
              },
              "metadata": {
                  "type": "object",
                  "properties": {
                      "parent_id": {
                          "type": "text"
                      },
                      "page_number": {
                          "type": "integer"
                      },
                      "is_continuation": {
                         "type": "boolean"
                      },
                      "orig_elements": {
                         "type": "text"
                      }
                  }
              }
          }
      }
  }
  

  See also:

  • Mappings and field types
  • Explicit mapping
  • Dynamic mapping
  • Unstructured document elements and metadata

• If you’re using basic authentication to the instance, the user’s name and password.

• If you’re using certificates for authentication instead:

  • The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA.
  • The path to the combined private key and certificate file, or
  • The paths to the separate private key and certificate file.

  To learn more, see:

  • For AWS OpenSearch instances, see Tutorial: Configure a domain with the internal user database and HTTP basic authentication and Creating a custom endpoint for Amazon OpenSearch Service.
  • For local OpenSearch instances, see Authentication backends, HTTP basic authentication, and Client certificate authentication.

The OpenSearch connector dependencies:

pip install "unstructured-ingest[opensearch]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

• OPENSEARCH_HOST - The hostname and port number, defined as <hostname>:<port-number> and represented by --hosts (CLI) or hosts (Python).
• OPENSEARCH_INDEX_NAME - The name of the search index, represented by --index-name (CLI) or index_name (Python).

If you’re using basic authentication to the instance:

• OPENSEARCH_USERNAME - The user’s name, represented by --username (CLI) or username (Python).
• OPENSEARCH_PASSWORD - The user’s password, represented by --password (CLI) or password (Python).

If you’re using certificates for authentication instead:

• OPENSEARCH_CA_CERTS - The path to the Certificate Authority (CA) bundle, if you use intermediate CAs with your root CA. This is represented by --ca-certs (CLI) or ca_certs (Python).
• OPENSEARCH_CLIENT_CERT - The path to the combined private key and certificate file, or the path to just the certificate file. This is represented by --client-cert (CLI) or client_cert (Python).
• OPENSEARCH_CLIENT_KEY - The path to the private key file, if OPENSEARCH_CLIENT_CERT refers to just the certificate file. This is represented by --client-key (CLI) or client_key (Python).

Additional related settings include:

• --use-ssl (CLI) or use_ssl=True (Python) to use SSL for the connection.
• --verify-certs (CLI) or verify_certs=True (Python) to verify SSL certificates.
• --ssl-show-warn (CLI) or ssl_show_warn=True (Python) to show a warning when verifying SSL certificates is disabled.

These environment variables:

• UNSTRUCTURED_API_KEY - Your Unstructured API key value.
• UNSTRUCTURED_API_URL - Your Unstructured API URL.

Now call the Unstructured CLI or Python SDK. The source connector can be any of the ones supported. This example uses the local source connector: